Demystifying the Resilience of Large Language Model Inference: An End-to-End Perspective.

Shiyang Chen 0004 Rutgers University, New Brunswick, NJ, USA Stevens Institute of Technology, Hoboken, NJ, USA https://sciprofiles.com/profile/2620903 https://orcid.org/0000-0003-2626-7865 https://www.wikidata.org/entity/Q130941145 Shiyang Chen Shiyang Chen 0001 Georgia Institute of Technology, Atlanta, GA, USA Emory University, Department of Biomedical Engineering, Atlanta, GA, USA Shiyang Chen 0002 Shanghai Jiao Tong University, Department of Electronic Engineering, China Shiyang Chen 0003 Beihang University, School of Electronic and Information Engineering, Beijing, China Shiyang Chen 0005 Peking University, School of Electronics Engineering and Computer Science, Beijing, China Shiyang Chen 0006 Anhui University, School of Computer Science and Technology, Hefei, China Shiyang Chen 0007 Chinese Academy of Sciences, Institute of Electronics, Key Laboratory of Technology in Geo-Spatial Information Processing and Application System, Beijing, China Shiyang Chen 0008 University of Michigan, Department of Mechanical Engineering, Ann Arbor, MI, USA https://shiyangchen1991.github.io/ https://scholar.google.com/citations?user=RKfw7fQAAAAJ Shiyang Chen 0009 Texas A&M University, College of Engineering, College Station, TX, USA https://ieeexplore.ieee.org/author/37085853679 https://orcid.org/0009-0003-7099-3511 Shiyang Chen 0010 China Securities Co., Ltd., Information Technology Department, China Shiyang Chen 0011 Chinese Academy of Agricultural Sciences, Institute of Agricultural Resources and Regional Planning, Beijing, China Shiyang Chen 0012 University of Hong Kong, Hong Kong Shiyang Chen 0013 Charles III University of Madrid, Spain Yu Sun Zachary Coalson Shiyang Chen 0004 Hang Liu 0001 Zhao Zhang 0007 Sanghyun Hong 0001 Bo Fang 0002 Lishan Yang 0001 Demystifying the Resilience of Large Language Model Inference: An End-to-End Perspective. 1127-1144 2025 SC https://doi.org/10.1145/3712285.3759803 conf/sc/2025 db/conf/sc/sc2025.html#SunC0L000Y25

Shiyang Chen 0004 Rain Jiang Dezhi Yu Jinlai Xu Mengyuan Chao Fanlong Meng Chenyu Jiang Wei Xu Hang Liu 0001 KVDirect: Distributed Disaggregated LLM Inference. 2025 January abs/2501.14743 CoRR https://doi.org/10.48550/arXiv.2501.14743 db/journals/corr/corr2501.html#abs-2501-14743 streams/journals/corr

Shiyang Chen 0004 Xiang Song Vasiloudis Theodore Hang Liu 0001 Deal: Distributed End-to-End GNN Inference for All Nodes. 2025 March abs/2503.02960 CoRR https://doi.org/10.48550/arXiv.2503.02960 db/journals/corr/corr2503.html#abs-2503-02960 streams/journals/corr

Hongwei Chen Shiyang Chen 0004 Joshua J. Turner Adrian E. Feiguin Kernel fusion in atomistic spin dynamics simulations on Nvidia GPUs using tensor core. 102357 2024 81 J. Comput. Sci. https://doi.org/10.1016/j.jocs.2024.102357 db/journals/jocs/jocs81.html#ChenCTF24

Chengying Huan Yongchao Liu 0004 Heng Zhang 0005 Shuaiwen Song Santosh Pandey 0001 Shiyang Chen 0004 Xiangfei Fang Yue Jin Baptiste Lepers Yanjun Wu Hang Liu 0001 <i>TEA+</i>: A Novel Temporal Graph Random Walk Engine with Hybrid Storage Architecture. 37 2024 June 21 ACM Trans. Archit. Code Optim. 2 https://doi.org/10.1145/3652604 https://www.wikidata.org/entity/Q130941149 db/journals/taco/taco21.html#HuanLZSPCFJLWL24 streams/journals/taco

Chengying Huan Yongchao Liu 0004 Heng Zhang 0005 Hang Liu 0001 Shiyang Chen 0004 Shuaiwen Leon Song Yanjun Wu TeGraph+: Scalable Temporal Graph Processing Enabling Flexible Edge Modifications. 1469-1487 2024 August 35 IEEE Trans. Parallel Distributed Syst. 8 https://doi.org/10.1109/TPDS.2024.3393914 db/journals/tpds/tpds35.html#HuanLZLCSW24 streams/journals/tpds

Haojun Xia Zhen Zheng Xiaoxia Wu Shiyang Chen 0004 Zhewei Yao Stephen Youn Arash Bakhtiari Michael Wyatt Donglin Zhuang Zhongzhu Zhou Olatunji Ruwase Yuxiong He Shuaiwen Leon Song Quant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric Algorithm-System Co-Design on Modern GPUs. 699-713 2024 USENIX ATC https://www.usenix.org/conference/atc24/presentation/xia conf/usenix/2024 db/conf/usenix/usenix2024.html#XiaZWCYYBWZZRHS24

Haojun Xia Zhen Zheng Xiaoxia Wu Shiyang Chen 0004 Zhewei Yao Stephen Youn Arash Bakhtiari Michael Wyatt Donglin Zhuang Zhongzhu Zhou Olatunji Ruwase Yuxiong He Shuaiwen Leon Song FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design. 2024 abs/2401.14112 CoRR https://doi.org/10.48550/arXiv.2401.14112 db/journals/corr/corr2401.html#abs-2401-14112

Zachary Coalson Jeonghyun Woo Shiyang Chen 0004 Yu Sun Lishan Yang 0001 Prashant J. Nair Bo Fang 0002 Sanghyun Hong 0001 PrisonBreak: Jailbreaking Large Language Models with Fewer Than Twenty-Five Targeted Bit-flips. 2024 abs/2412.07192 CoRR https://doi.org/10.48550/arXiv.2412.07192 db/journals/corr/corr2412.html#abs-2412-07192 streams/journals/corr

Yifei Wang 0002 Shiyang Chen 0004 Guobin Chen Ethan Shurberg Hang Liu 0001 Pengyu Hong Motif-Based Graph Representation Learning with Application to Chemical Molecules. 8 2023 March 10 Informatics 1 https://doi.org/10.3390/informatics10010008 db/journals/informatics/informatics10.html#WangCCSLH23

Wang Feng Shiyang Chen 0004 Hang Liu 0001 Yuede Ji PeeK: A Prune-Centric Approach for K Shortest Path Computation. 18:1-18:14 2023 SC https://doi.org/10.1145/3581784.3607110 conf/sc/2023 db/conf/sc/sc2023.html#FengCLJ23 Shiyang Chen 0004 Da Zheng 0004 Caiwen Ding Chengying Huan Yuede Ji Hang Liu 0001 TANGO: re-thinking quantization for graph neural network training on GPUs. 38:1-38:14 2023 SC https://doi.org/10.1145/3581784.3607037 https://www.wikidata.org/entity/Q130953679 conf/sc/2023 db/conf/sc/sc2023.html#ChenZDHJL23

Shiyang Chen 0004 Da Zheng 0004 Caiwen Ding Chengying Huan Yuede Ji Hang Liu 0001 Tango: rethinking quantization for graph neural network training on GPUs. 2023 abs/2308.00890 CoRR https://doi.org/10.48550/arXiv.2308.00890 db/journals/corr/corr2308.html#abs-2308-00890

Shuaiwen Leon Song Bonnie Kruft Minjia Zhang Conglong Li Shiyang Chen 0004 Chengming Zhang 0006 Masahiro Tanaka Xiaoxia Wu Jeff Rasley Ammar Ahmad Awan Connor Holmes Martin Cai Adam Ghanem Zhongzhu Zhou Yuxiong He Pete Luferenko Divya Kumar Jonathan A. Weyn Ruixiong Zhang Sylwester Klocek Volodymyr Vragov Mohammed AlQuraishi Gustaf Ahdritz Christina Floristean Cristina Negri Rao Kotamarthi Venkatram Vishwanath Arvind Ramanathan Sam Foreman Kyle Hippe Troy Arcomano Romit Maulik Maxim Zvyagin Alexander Brace Bin Zhang Cindy Orozco Bohorquez Austin Clyde Bharat Kale Danilo Perez-Rivera Heng Ma Carla M. Mann Michael W. Irvin J. Gregory Pauloski Logan T. Ward Valérie Hayot-Sasson Murali Emani Zhen Xie Diangen Lin Maulik Shukla Ian T. Foster James J. Davis 0002 Michael E. Papka Thomas S. Brettin Prasanna Balaprakash Gina Tourassi John Gounley Heidi A. Hanson Thomas E. Potok Massimiliano Lupo Pasini Kate Evans Dan Lu 0001 Dalton D. Lunga Junqi Yin Sajal Dash Feiyi Wang Mallikarjun Shankar Isaac Lyngaas Xiao Wang 0004 Guojing Cong Pei Zhang Ming Fan Siyan Liu Adolfy Hoisie Shinjae Yoo Yihui Ren 0001 William Tang 0002 Kyle Felker Alexey Svyatkovskiy Hang Liu 0001 Ashwin M. Aji Angela Dalton Michael J. Schulte Karl W. Schulz Yuntian Deng Weili Nie Josh Romero Christian Dallago Arash Vahdat Chaowei Xiao Thomas Gibbs Anima Anandkumar Rick Stevens DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies. 2023 abs/2310.04610 CoRR https://doi.org/10.48550/arXiv.2310.04610 db/journals/corr/corr2310.html#abs-2310-04610

Xiaoxia Wu Haojun Xia Stephen Youn Zhen Zheng Shiyang Chen 0004 Arash Bakhtiari Michael Wyatt Reza Yazdani Aminabadi Yuxiong He Olatunji Ruwase Leon Song Zhewei Yao ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks. 2023 abs/2312.08583 CoRR https://doi.org/10.48550/arXiv.2312.08583 db/journals/corr/corr2312.html#abs-2312-08583

Shaoyi Huang Dongkuan Xu Ian En-Hsu Yen Yijue Wang Sung-En Chang Bingbing Li Shiyang Chen 0004 Mimi Xie Sanguthevar Rajasekaran Hang Liu 0001 Caiwen Ding Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. 190-200 2022 ACL (1) https://doi.org/10.18653/v1/2022.acl-long.16 https://aclanthology.org/2022.acl-long.16 https://www.wikidata.org/entity/Q131458247 conf/acl/2022-1 db/conf/acl/acl2022-1.html#HuangXYWCLCXRLD22 Hongwu Peng Shaoyi Huang Shiyang Chen 0004 Bingbing Li Tong Geng Ang Li 0006 Weiwen Jiang Wujie Wen Jinbo Bi Hang Liu 0001 Caiwen Ding A length adaptive algorithm-hardware co-design of transformer on FPGA through sparse attention and dynamic pipelining. 1135-1140 2022 DAC https://doi.org/10.1145/3489517.3530585 https://www.wikidata.org/entity/Q130827457 conf/dac/2022 db/conf/dac/dac2022.html#PengHCLGLJWBLD22

Hongwu Peng Shaoyi Huang Shiyang Chen 0004 Bingbing Li Tong Geng Ang Li 0006 Weiwen Jiang Wujie Wen Jinbo Bi Hang Liu 0001 Caiwen Ding A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipelining. 2022 abs/2208.03646 CoRR https://doi.org/10.48550/arXiv.2208.03646 db/journals/corr/corr2208.html#abs-2208-03646

Yifei Wang 0002 Shiyang Chen 0004 Guobin Chen Ethan Shurberg Hang Liu 0001 Pengyu Hong Motif-based Graph Representation Learning with Application to Chemical Molecules. 2022 abs/2208.04529 CoRR https://doi.org/10.48550/arXiv.2208.04529 db/journals/corr/corr2208.html#abs-2208-04529

Shaoyi Huang Shiyang Chen 0004 Hongwu Peng Daniel Manu Zhenglun Kong Geng Yuan Lei Yang 0018 Shusen Wang Hang Liu 0001 Caiwen Ding HMC-TRAN: A Tensor-core Inspired Hierarchical Model Compression for Transformer-based DNNs on GPU. 169-174 2021 ACM Great Lakes Symposium on VLSI https://doi.org/10.1145/3453688.3461740 conf/glvlsi/2021 db/conf/glvlsi/glvlsi2021.html#HuangCPMKYYW0D21 Hongwu Peng Shiyang Chen 0004 Zhepeng Wang 0001 Junhuan Yang Scott A. Weitze Tong Geng Ang Li 0006 Jinbo Bi Minghu Song Weiwen Jiang Hang Liu 0001 Caiwen Ding Optimizing FPGA-based Accelerator Design for Large-Scale Molecular Similarity Search (Special Session Paper). 1-7 2021 ICCAD https://doi.org/10.1109/ICCAD51958.2021.9643528 conf/iccad/2021 db/conf/iccad/iccad2021.html#PengCWYWGLBSJLD21 Shiyang Chen 0004 Shaoyi Huang Santosh Pandey 0001 Bingbing Li Guang R. Gao Long Zheng 0001 Caiwen Ding Hang Liu 0001 E.T.: re-thinking self-attention for transformer models on GPUs. 25 2021 SC https://doi.org/10.1145/3458817.3476138 conf/sc/2021 db/conf/sc/sc2021.html#ChenHPLG0D021

Hongwu Peng Shiyang Chen 0004 Zhepeng Wang 0001 Junhuan Yang Scott A. Weitze Tong Geng Ang Li 0006 Jinbo Bi Minghu Song Weiwen Jiang Hang Liu 0001 Caiwen Ding Optimizing FPGA-based Accelerator Design for Large-Scale Molecular Similarity Search. 2021 abs/2109.06355 CoRR https://arxiv.org/abs/2109.06355 db/journals/corr/corr2109.html#abs-2109-06355

Shaoyi Huang Dongkuan Xu Ian En-Hsu Yen Sung-En Chang Bingbing Li Shiyang Chen 0004 Mimi Xie Hang Liu 0001 Caiwen Ding Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm. 2021 abs/2110.08190 CoRR https://arxiv.org/abs/2110.08190 db/journals/corr/corr2110.html#abs-2110-08190

Gustaf Ahdritz Ashwin M. Aji Mohammed AlQuraishi Anima Anandkumar Troy Arcomano Ammar Ahmad Awan Arash Bakhtiari Prasanna Balaprakash Jinbo Bi Cindy Orozco Bohorquez Alex BraceAlexander Brace Thomas S. Brettin Martin Cai Sung-En Chang Mengyuan Chao Guobin Chen Hongwei Chen Austin Clyde Zachary Coalson Guojing Cong Christian Dallago Angela Dalton Sajal Dash James J. Davis 0002 Yuntian Deng Caiwen Ding Murali Emani Kate Evans Ming Fan Bo Fang 0002 Xiangfei Fang Adrian E. Feiguin Kyle Felker Wang Feng Christina Floristean Sam Foreman Ian T. Foster Guang R. Gao Tong Geng Adam Ghanem Thomas Gibbs John Gounley Heidi A. Hanson Valérie Hayot-Sasson Yuxiong He Kyle Hippe Adolfy Hoisie Connor Holmes Pengyu Hong Sanghyun Hong 0001 Chengying Huan Shaoyi Huang Michael W. Irvin Yuede Ji Chenyu Jiang Rain Jiang Weiwen Jiang Yue Jin Bharat Kale Sylwester Klocek Zhenglun Kong Rao Kotamarthi Bonnie Kruft Divya Kumar Baptiste Lepers Ang Li 0006 Bingbing Li Conglong Li Diangen Lin Hang Liu 0001 Siyan Liu Yongchao Liu 0004 Dan Lu 0001 Pete Luferenko Dalton D. Lunga Isaac Lyngaas Heng Ma Carla M. Mann Daniel Manu Romit Maulik Fanlong Meng Prashant J. Nair Cristina Negri Weili Nie Santosh Pandey 0001 Michael E. Papka Massimiliano Lupo Pasini J. Gregory Pauloski Hongwu Peng Danilo Perez-Rivera Thomas E. Potok Sanguthevar Rajasekaran Arvind Ramanathan Jeff Rasley Yihui Ren 0001 Josh Romero Olatunji Ruwase Michael J. Schulte Karl W. Schulz Mallikarjun Shankar Maulik Shukla Ethan Shurberg Leon Song Minghu Song Shuaiwen SongShuaiwen Leon Song Xiang Song Rick L. StevensRick Stevens Yu Sun Alexey Svyatkovskiy Masahiro Tanaka William Tang 0002 Vasiloudis Theodore Gina Tourassi Joshua J. Turner Arash Vahdat Venkatram Vishwanath Volodymyr Vragov Feiyi Wang Shusen Wang Xiao Wang 0004 Yifei Wang 0002 Yijue Wang Zhepeng Wang 0001 Logan T. Ward Scott WeitzeScott A. Weitze Wujie Wen Jonathan A. Weyn Jeonghyun Woo Xiaoxia Wu Yanjun Wu Michael Wyatt Haojun Xia Chaowei Xiao Mimi Xie Zhen Xie Dongkuan Xu Jinlai Xu Wei Xu Junhuan Yang Lei Yang 0018 Lishan Yang 0001 Zhewei Yao Reza YazdaniReza Yazdani Aminabadi Ian En-Hsu Yen Junqi Yin Shinjae Yoo Stephen Youn Dezhi Yu Geng Yuan Bin Zhang Chengming Zhang 0006 Heng Zhang 0005 Minjia Zhang Pei Zhang Ruixiong Zhang Zhao Zhang 0007 Da Zheng 0004 Long Zheng 0001 Zhen Zheng Zhongzhu Zhou Donglin Zhuang Maxim Zvyagin