Compound AI Architectures

Last Updated 18 June, 2026

by David Spuler, Ph.D.

Compound AI: Book Excerpts and Blog Articles

Free online book excerpts with full text chapters online and free PDF downloads, and the Aussie AI blog, including related articles:

David Spuler, Michael Sharpe, 2025, Architectures of AI Projects, Chapter 12, in "Generative AI Applications", https://www.aussieai.com/book/ai-apps-book-12-architectures-project

Research on Compound AI Architectures

Research papers include:

Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, Azalia Mirhoseini, 31 Jul 2024, Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, https://arxiv.org/abs/2407.21787 (Generating multiple answers by repeated inference queries, and then using a verifier to choose the best one, which is shown to greatly increase overall accuracy.)
Xi Wang, Procheta Sen, Ruizhe Li, Emine Yilmaz, 31 Jul 2024, Adaptive Retrieval-Augmented Generation for Conversational Systems, https://arxiv.org/abs/2407.21712 (Deciding whether or not to include a RAG external data request in the inference of a chatbot in a multi-turn conversation.)
Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, Ali Ghodsi, Feb 18, 2024, The Shift from Models to Compound AI Systems, https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/
Jared Quincy Davis, Boris Hanin, Lingjiao Chen, Peter Bailis, Ion Stoica, Matei Zaharia, 23 Jul 2024, Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Design, https://www.arxiv.org/abs/2407.16831
Sherry Ruan, Tian Zhao, 28 May 2024, JungleGPT: Designing and Optimizing Compound AI Systems for E-Commerce, https://arxiv.org/abs/2407.00038
Cognine, 2024, Why 2024 is the Year of AI Agents and Compound AI Systems? https://cognine.com/why-2024-is-the-year-of-ai-agents-and-compound-ai-systems/
Sean Sheng and Sherlock Xu, August 15, 2024, A Guide to Compound AI Systems, https://www.bentoml.com/blog/a-guide-to-compound-ai-systems
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 6 Jun 2024 (v2), SGLang: Efficient Execution of Structured Language Model Programs, https://arxiv.org/abs/2312.07104 https://github.com/sgl-project/sglang
An Efficient Network Orchestrator for Distributed Compound Language Model Systems Muhammad Shahir Abdurrahman, Stanford University, Stanford, California, USA, https://www.scs.stanford.edu/24sp-cs244b/projects/An_Efficient_Network_Orchestrator_for_Distributed_Compound_Language_Model_Systems.pdf
Melissa Malec, June 5, 2024, AI Orchestration Explained: The What, Why & How for 2024, https://hatchworks.com/blog/gen-ai/ai-orchestration/
Yanxi Chen, Yaliang Li, Bolin Ding, Jingren Zhou, 20 Jul 2024, On the Design and Analysis of LLM-Based Algorithms, https://arxiv.org/abs/2407.14788 https://github.com/modelscope/agentscope/tree/main/examples/paper_llm_based_algorithm
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, James Zou, 4 Jun 2024 (v2), Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systems, https://arxiv.org/abs/2403.02419
Latent Space, Nov 2024, Why Compound AI + Open Source will beat Closed AI, https://www.latent.space/p/fireworks
Gohar Irfan Chaudhry, Esha Choukse, Íñigo Goiri, Rodrigo Fonseca, Adam Belay, Ricardo Bianchini, 29 Jan 2025 (v2), Towards Resource-Efficient Compound AI Systems, https://arxiv.org/abs/2501.16634
Lingjiao Chen, Jared Quincy Davis, Boris Hanin, Peter Bailis, Matei Zaharia, James Zou, Ion Stoica, 20 Feb 2025, Optimizing Model Selection for Compound AI Systems, https://arxiv.org/abs/2502.14815
Rajeshkumar Bambhaniya, Abhimanyu ; Wu, Hanjiang ; Subramanian, Suvinay ; Srinivasan, Sudarshan ; Kundu, Souvik ; Yazdanbakhsh, Amir ; Elavazhagan, Midhilesh ; Kumar, Madhu ; Krishna, Tushar, April 2025, Understanding and Optimizing Multi-Stage AI Inference Pipelines, https://ui.adsabs.harvard.edu/abs/2025arXiv250409775R/abstract https://arxiv.org/abs/2504.09775
OnlyCFO, Apr 29, 2025, Bullish: Vertical & Compound Software: In a world of AI, companies need to be more multi-product and vertical to win, https://www.onlycfo.io/p/bullish-vertical-and-compound-software
Tomasz Tunguz, Jul 17, 2025, Hidden Technical Debt in AI, https://tomtunguz.com/hidden-technical-debt-in-ai/
Yang Liu, Bingjie Yan, Tianyuan Zou, Jianqing Zhang, Zixuan Gu, Jianbing Ding, Xidong Wang, Jingyi Li, Xiaozhou Ye, Ye Ouyang, Qiang Yang, Ya-Qin Zhang, 24 Apr 2025, Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks, https://arxiv.org/abs/2504.17421
Marc Brooker, Aug 2025, LLMs as Parts of Systems, https://brooker.co.za/blog/2025/08/12/llms-as-components.html
Deepti Raghavan, Keshav Santhanam, Muhammad Shahir Rahman, Nayani Modugula, Luis Gaspar Schroeder, Maximilien Cura, Houjun Liu, Pratiksha Thaker, Philip Levis, Matei Zaharia, 22 Jul 2025, Alto: Orchestrating Distributed Compound AI Systems with Nested Ancestry, https://arxiv.org/abs/2403.04311
Soheil Radfar, Faezeh Maghsoodifar, Hamed Moftakhari and Hamid Moradkhani, 20 Jul 2025, Integrating Newton's Laws with deep learning for enhanced physics-informed compound flood modelling, https://arxiv.org/abs/2507.15021
Hongzhi Zhang, Zhonglie Liu, Kun Meng, Jiameng Chen, Jia Wu, Bo Du, Di Lin, Yan Che, Wenbin Hu, 28 Jul 2025, Zero-Shot Learning with Subsequence Reordering Pretraining for Compound-Protein Interaction, https://arxiv.org/abs/2507.20925
Nguyen Manh Son, Pham Huu Vang, Nguyen Thi Dung, Nguyen Manh Ha. Ta Thi Thao, Tran Thi Thu Thuy, Phan Minh Giang, 13 Aug 2025, In silico study on the cytotoxicity against Hela cancer cells of xanthones bioactive compounds from Garcinia cowa: QSAR based on Graph Deep Learning, Network Pharmacology, and Molecular Docking, https://arxiv.org/abs/2508.10117
Wonjun Yi, Wonho Jung, Hyeonuk Nam, Kangmin Jang, Yong-Hwa Park, 8 Sep 2025, Multi-output Classification using a Cross-talk Architecture for Compound Fault Diagnosis of Motors in Partially Labeled Condition, https://arxiv.org/abs/2505.24001
Jonathan Adam Rico, Nagarajan Raghavan, and Senthilnath Jayavelu, 19 Sep 2025, Compound Fault Diagnosis for Train Transmission Systems Using Deep Learning with Fourier-enhanced Representation, https://arxiv.org/abs/2504.07155
Md Ashad Alam and Md Amanullah, 22 Oct 2025, Artificial Intelligence Powered Identification of Potential Antidiabetic Compounds in Ficus religiosa, https://arxiv.org/abs/2510.19867
Eduard Popescu and Adrian Groza and Andreea Cernat, 26 Oct 2025, Combining Deep Learning and Explainable AI for Toxicity Prediction of Chemical Compounds, https://arxiv.org/abs/2510.22572
Zanyu Shi, Yang Wang, Pathum Weerawarna, Jie Zhang, Timothy Richardson, Yijie Wang, Kun Huang, 7 Oct 2025, Structure-Aware Compound-Protein Affinity Prediction via Graph Neural Network with Group Lasso Regularization, https://arxiv.org/abs/2507.03318
Pawel Trajdos and Marek Kurzynski, 24 Sep 2025, A Compound Classification System Based on Fuzzy Relations Applied to the Noise-Tolerant Control of a Bionic Hand via EMG Signal Recognition, https://arxiv.org/abs/2509.20523
Snehal Raj, Brian Coyle, 5 Oct 2025, QuIC: Quantum-Inspired Compound Adapters for Parameter Efficient Fine-Tuning, https://arxiv.org/abs/2502.06916
Shirley Wu, Parth Sarthi, Shiyu Zhao, Aaron Lee, Herumb Shandilya, Adrian Mladenic Grobelnik, Nurendra Choudhary, Eddie Huang, Karthik Subbian, Linjun Zhang, Diyi Yang, James Zou, Jure Leskovec, 5 Oct 2025, Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards, https://arxiv.org/abs/2507.03041
Yu-Ang Lee, Guan-Ting Yi, Mei-Yi Liu, Jui-Chao Lu, Guan-Bo Yang, Yun-Nung Chen, 7 Oct 2025, Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions, https://arxiv.org/abs/2506.08234
Contrary Research, Dec 2025, The Age of Composable Software, https://research.contrary.com/report/the-age-of-composable-software
David Spuler, Michael Sharpe, 2025, Architectures of AI Projects, Chapter 12, in "Generative AI Applications", https://www.aussieai.com/book/ai-apps-book-12-architectures-project
Yuhan Liu, Yuyang Huang, Jiayi Yao, Shaoting Feng, Zhuohan Gu, Kuntai Du, Hanchen Li, Yihua Cheng, and Junchen Jiang, Shan Lu, Madan Musuvathi, and Esha Choukse, May 2026, DroidSpeak: KV Cache Sharing Across Fine-tuned Model Variants, Proceedings of the 23rd USENIX Symposium on Networked Systems Design and Implementation. May 4–6, 2026 • Renton, WA, USA, https://www.usenix.org/system/files/nsdi26-liu-yuhan.pdf https://www.usenix.org/conference/nsdi26/presentation/liu-yuhan (Sharing prefix KV caches from different fine-tuned LLMs, with partial layerwise recomputation.)
OpenRouter, Jun 13, 2026, Introducing the Fusion API, the smartest compound model in the market, https://threadreaderapp.com/thread/2065856853989270011.html