{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,4]],"date-time":"2026-07-04T16:40:36Z","timestamp":1783183236513,"version":"3.54.6"},"reference-count":110,"publisher":"Association for Computing Machinery (ACM)","issue":"4","license":[{"start":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T00:00:00Z","timestamp":1721347200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Graph."],"published-print":{"date-parts":[[2024,7,19]]},"abstract":"<jats:p>\n            In this work, we present\n            <jats:italic>Semantic Gesticulator<\/jats:italic>\n            , a novel framework designed to synthesize realistic gestures accompanying speech with strong semantic correspondence. Semantically meaningful gestures are crucial for effective non-verbal communication, but such gestures often fall within the long tail of the distribution of natural human motion. The sparsity of these movements makes it challenging for deep learning-based systems, trained on moderately sized datasets, to capture the relationship between the movements and the corresponding speech semantics. To address this challenge, we develop a generative retrieval framework based on a large language model. This framework efficiently retrieves suitable semantic gesture candidates from a motion library in response to the input speech. To construct this motion library, we summarize a comprehensive list of commonly used semantic gestures based on findings in linguistics, and we collect a high-quality motion dataset encompassing both body and hand movements. We also design a novel GPT-based model with strong generalization capabilities to audio, capable of generating high-quality gestures that match the rhythm of speech. Furthermore, we propose a semantic alignment mechanism to efficiently align the retrieved semantic gestures with the GPT's output, ensuring the naturalness of the final animation. Our system demonstrates robustness in generating gestures that are rhythmically coherent and semantically explicit, as evidenced by a comprehensive collection of examples. User studies confirm the quality and human-likeness of our results, and show that our system outperforms state-of-the-art systems in terms of semantic appropriateness by a clear margin. We will release the code and dataset for academic research.\n          <\/jats:p>","DOI":"10.1145\/3658134","type":"journal-article","created":{"date-parts":[[2024,7,19]],"date-time":"2024-07-19T14:47:57Z","timestamp":1721400477000},"page":"1-17","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":38,"title":["Semantic Gesticulator: Semantics-Aware Co-Speech Gesture Synthesis"],"prefix":"10.1145","volume":"43","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2029-7118","authenticated-orcid":false,"given":"Zeyi","family":"Zhang","sequence":"first","affiliation":[{"name":"School of EECS, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7418-1014","authenticated-orcid":false,"given":"Tenglong","family":"Ao","sequence":"additional","affiliation":[{"name":"School of Computer Science, Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0009-4632-7934","authenticated-orcid":false,"given":"Yuyao","family":"Zhang","sequence":"additional","affiliation":[{"name":"Renmin University of China, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6239-571X","authenticated-orcid":false,"given":"Qingzhe","family":"Gao","sequence":"additional","affiliation":[{"name":"Shandong University, Beijing, China"},{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-1934-2332","authenticated-orcid":false,"given":"Chuan","family":"Lin","sequence":"additional","affiliation":[{"name":"Peking University, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4702-036X","authenticated-orcid":false,"given":"Baoquan","family":"Chen","sequence":"additional","affiliation":[{"name":"School of Intelligence Science Technology, Peking University, Beijing, China"},{"name":"State Key Laboratory of General Artificial Intelligence, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2280-6817","authenticated-orcid":false,"given":"Libin","family":"Liu","sequence":"additional","affiliation":[{"name":"School of Intelligence Science Technology, Peking University, Beijing, China"},{"name":"State Key Laboratory of General Artificial Intelligence, Beijing, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2024,7,19]]},"reference":[{"key":"e_1_2_2_1_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.13946"},{"key":"e_1_2_2_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/3592458"},{"key":"e_1_2_2_3_1","unstructured":"Alibaba. 2009. Alibaba Cloud Automatic Speech Recognition. Accessed: 2023-12-15."},{"key":"e_1_2_2_4_1","unstructured":"Rohan Anil Andrew M. Dai Orhan Firat Melvin Johnson Dmitry Lepikhin Alexandre Passos Siamak Shakeri Emanuel Taropa Paige Bailey Zhifeng Chen Eric Chu Jonathan H. Clark Laurent El Shafey Yanping Huang Kathy Meier-Hellstern Gaurav Mishra Erica Moreira Mark Omernick Kevin Robinson Sebastian Ruder Yi Tay Kefan Xiao Yuanzhong Xu Yujing Zhang Gustavo Hernandez Abrego Junwhan Ahn Jacob Austin Paul Barham Jan Botha James Bradbury Siddhartha Brahma Kevin Brooks Michele Catasta Yong Cheng Colin Cherry Christopher A. Choquette-Choo Aakanksha Chowdhery Cl\u00e9ment Crepy Shachi Dave Mostafa Dehghani Sunipa Dev Jacob Devlin Mark D\u00edaz Nan Du Ethan Dyer Vlad Feinberg Fangxiaoyu Feng Vlad Fienber Markus Freitag Xavier Garcia Sebastian Gehrmann Lucas Gonzalez Guy Gur-Ari Steven Hand Hadi Hashemi Le Hou Joshua Howland Andrea Hu Jeffrey Hui Jeremy Hurwitz Michael Isard Abe Ittycheriah Matthew Jagielski Wenhao Jia Kathleen Kenealy Maxim Krikun Sneha Kudugunta Chang Lan Katherine Lee Benjamin Lee Eric Li Music Li Wei Li YaGuang Li Jian Li Hyeontaek Lim Hanzhao Lin Zhongtao Liu Frederick Liu Marcello Maggioni Aroma Mahendru Joshua Maynez Vedant Misra Maysam Moussalem Zachary Nado John Nham Eric Ni Andrew Nystrom Alicia Parrish Marie Pellat Martin Polacek Alex Polozov Reiner Pope Siyuan Qiao Emily Reif Bryan Richter Parker Riley Alex Castro Ros Aurko Roy Brennan Saeta Rajkumar Samuel Renee Shelby Ambrose Slone Daniel Smilkov David R. So Daniel Sohn Simon Tokumine Dasha Valter Vijay Vasudevan Kiran Vodrahalli Xuezhi Wang Pidong Wang Zirui Wang Tao Wang John Wieting Yuhuai Wu Kelvin Xu Yunhan Xu Linting Xue Pengcheng Yin Jiahui Yu Qiao Zhang Steven Zheng Ce Zheng Weikang Zhou Denny Zhou Slav Petrov and Yonghui Wu. 2023. PaLM 2 Technical Report. arXiv:2305.10403 [cs.CL]"},{"key":"e_1_2_2_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/3550454.3555435"},{"key":"e_1_2_2_6_1","doi-asserted-by":"publisher","DOI":"10.1145\/3592097"},{"key":"e_1_2_2_7_1","volume-title":"Technical Report MSR-TR-2000-65. 8 pages. https:\/\/www.microsoft.com\/en-us\/research\/publication\/constrained-k-means-clustering\/","author":"Bennett K.P.","year":"2000","unstructured":"K.P. Bennett, P.S. Bradley, and A. Demiriz. 2000. Constrained K-Means Clustering. Technical Report MSR-TR-2000-65. 8 pages. https:\/\/www.microsoft.com\/en-us\/research\/publication\/constrained-k-means-clustering\/"},{"key":"e_1_2_2_8_1","volume-title":"Sebastian Riedel, and Fabio Petroni.","author":"Bevilacqua Michele","year":"2022","unstructured":"Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Wen tau Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive Search Engines: Generating Substrings as Document Identifiers. arXiv:2204.10628 [cs.CL]"},{"key":"e_1_2_2_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/3474085.3475223"},{"key":"e_1_2_2_10_1","doi-asserted-by":"publisher","DOI":"10.1109\/VR50410.2021.00037"},{"key":"e_1_2_2_11_1","unstructured":"Nicola De Cao Gautier Izacard Sebastian Riedel and Fabio Petroni. 2021. Autoregressive Entity Retrieval. arXiv:2010.00904 [cs.CL]"},{"key":"e_1_2_2_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2017.143"},{"key":"e_1_2_2_13_1","doi-asserted-by":"publisher","DOI":"10.1145\/192161.192272"},{"key":"e_1_2_2_14_1","doi-asserted-by":"publisher","DOI":"10.1145\/383259.383315"},{"key":"e_1_2_2_15_1","doi-asserted-by":"crossref","unstructured":"Yupeng Chang Xu Wang Jindong Wang Yuan Wu Linyi Yang Kaijie Zhu Hao Chen Xiaoyuan Yi Cunxiang Wang Yidong Wang Wei Ye Yue Zhang Yi Chang Philip S. Yu Qiang Yang and Xing Xie. 2023. A Survey on Evaluation of Large Language Models. arXiv:2307.03109 [cs.CL]","DOI":"10.1145\/3641289"},{"key":"e_1_2_2_16_1","volume-title":"DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation. Preprint","author":"Chen Junming","year":"2023","unstructured":"Junming Chen, Yunfei Liu, Jianan Wang, Ailing Zeng, Yu Li, and Qifeng Chen. 2023. DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation. Preprint (2023)."},{"key":"e_1_2_2_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/3511808.3557271"},{"key":"e_1_2_2_18_1","volume-title":"Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion. arXiv preprint arXiv:2312.04466","author":"Chhatre Kiran","year":"2023","unstructured":"Kiran Chhatre, Radek Dan\u011b\u010dek, Nikos Athanasiou, Giorgio Becherini, Christopher Peters, Michael J Black, and Timo Bolkart. 2023. Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusion. arXiv preprint arXiv:2312.04466 (2023)."},{"key":"e_1_2_2_19_1","unstructured":"Paul Christiano Jan Leike Tom B. Brown Miljan Martic Shane Legg and Dario Amodei. 2023. Deep reinforcement learning from human preferences. arXiv:1706.03741 [stat.ML]"},{"key":"e_1_2_2_20_1","unstructured":"Credamo. 2017. Credamo: an online data survey platform. Accessed: 2023-12-15."},{"key":"e_1_2_2_21_1","unstructured":"DeepMotion. 2024. DeepMotion - AI Motion Capture & Body Tracking. Accessed: 2024-1-2."},{"key":"e_1_2_2_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3577190.3616117"},{"key":"e_1_2_2_23_1","volume-title":"Alec Radford, and Ilya Sutskever.","author":"Dhariwal Prafulla","year":"2020","unstructured":"Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. 2020. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341 (2020)."},{"key":"e_1_2_2_24_1","volume-title":"IVA '18 Proceedings of the 18th International Conference on Intelligent Virtual Agents. https:\/\/trinityspeechgesture.scss.tcd.ie","author":"Ferstl Ylva","year":"2018","unstructured":"Ylva Ferstl and Rachel McDonnell. 2018. IVA: Investigating the use of recurrent motion modelling for speech gesture generation. In IVA '18 Proceedings of the 18th International Conference on Intelligent Virtual Agents. https:\/\/trinityspeechgesture.scss.tcd.ie"},{"key":"e_1_2_2_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cag.2020.04.007"},{"key":"e_1_2_2_26_1","doi-asserted-by":"publisher","DOI":"10.1002\/cav.2016"},{"key":"e_1_2_2_27_1","volume-title":"Ges-GPT: Speech Gesture Synthesis With Text Parsing from GPT. arXiv preprint arXiv:2303.13013","author":"Gao Nan","year":"2023","unstructured":"Nan Gao, Zeyu Zhao, Zhi Zeng, Shuwu Zhang, and Dongdong Weng. 2023. Ges-GPT: Speech Gesture Synthesis With Text Parsing from GPT. arXiv preprint arXiv:2303.13013 (2023)."},{"key":"e_1_2_2_28_1","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.14734"},{"key":"e_1_2_2_29_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2019.00361"},{"key":"e_1_2_2_30_1","volume-title":"Michael Bi Mi, and Xinchao Wang","author":"Gong Kehong","year":"2023","unstructured":"Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Zihang Jiang, Xinxin Zuo, Michael Bi Mi, and Xinchao Wang. 2023. TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration. arXiv:2304.02419 [cs.CV]"},{"key":"e_1_2_2_31_1","doi-asserted-by":"publisher","DOI":"10.1145\/3528233.3530750"},{"key":"e_1_2_2_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472306.3478335"},{"key":"e_1_2_2_33_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2002.06.001"},{"key":"e_1_2_2_34_1","volume-title":"Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299","author":"Izacard Gautier","year":"2022","unstructured":"Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299 (2022)."},{"key":"e_1_2_2_35_1","volume-title":"C2G2: Controllable Co-speech Gesture Generation with Latent Diffusion Model. arXiv preprint arXiv:2308.15016","author":"Ji Longbin","year":"2023","unstructured":"Longbin Ji, Pengfei Wei, Yi Ren, Jinglin Liu, Chen Zhang, and Xiang Yin. 2023. C2G2: Controllable Co-speech Gesture Generation with Latent Diffusion Model. arXiv preprint arXiv:2308.15016 (2023)."},{"key":"e_1_2_2_36_1","unstructured":"Michael Kipp. 2004. Gesture Generation by Imitation: From Human Behavior to Computer Character Animation. Dissertation.com Boca Raton."},{"key":"e_1_2_2_37_1","unstructured":"Michael Kipp. 2005. Gesture generation by imitation: from human behavior to computer character animation. https:\/\/api.semanticscholar.org\/CorpusID:26271318"},{"key":"e_1_2_2_38_1","doi-asserted-by":"publisher","DOI":"10.1007\/11821830_17"},{"key":"e_1_2_2_39_1","doi-asserted-by":"publisher","DOI":"10.1145\/3382507.3418815"},{"key":"e_1_2_2_40_1","doi-asserted-by":"publisher","DOI":"10.1145\/3472306.3478333"},{"key":"e_1_2_2_41_1","volume-title":"Evaluating gesture-generation in a large-scale open challenge: The GENEA Challenge","author":"Kucherenko Taras","year":"2022","unstructured":"Taras Kucherenko, Pieter Wolfert, Youngwoo Yoon, Carla Viegas, Teodor Nikolov, Mihail Tsakov, and Gustav Eje Henter. 2023. Evaluating gesture-generation in a large-scale open challenge: The GENEA Challenge 2022. arXiv preprint arXiv:2303.08737 (2023)."},{"key":"e_1_2_2_42_1","volume-title":"Proceedings of the IEEE\/CVF International Conference on Computer Vision. 763--772","author":"Lee Gilwoo","year":"2019","unstructured":"Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S Srinivasa, and Yaser Sheikh. 2019. Talking with hands 16.2 m: A large-scale dataset of synchronized body-finger motion and audio for conversational motion analysis and synthesis. In Proceedings of the IEEE\/CVF International Conference on Computer Vision. 763--772."},{"key":"e_1_2_2_43_1","doi-asserted-by":"publisher","DOI":"10.1145\/1778765.1778861"},{"key":"e_1_2_2_44_1","doi-asserted-by":"publisher","DOI":"10.1145\/1618452.1618518"},{"key":"e_1_2_2_45_1","first-page":"9459","article-title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","volume":"33","author":"Lewis Patrick","year":"2020","unstructured":"Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\u00fcttler, Mike Lewis, Wen-tau Yih, Tim Rockt\u00e4schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459--9474.","journal-title":"Advances in Neural Information Processing Systems"},{"key":"e_1_2_2_46_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV48922.2021.01110"},{"key":"e_1_2_2_47_1","unstructured":"Yongqi Li Nan Yang Liang Wang Furu Wei and Wenjie Li. 2023. Multiview Identifiers Enhanced Generative Retrieval. arXiv:2305.16675 [cs.CL]"},{"key":"e_1_2_2_48_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01022"},{"key":"e_1_2_2_49_1","doi-asserted-by":"publisher","DOI":"10.1145\/3503161.3548400"},{"key":"e_1_2_2_50_1","volume-title":"Black","author":"Liu Haiyang","year":"2023","unstructured":"Haiyang Liu, Zihao Zhu, Giorgio Becherini, Yichen Peng, Mingyang Su, You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, and Michael J. Black. 2023. EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Masked Audio Gesture Modeling. arXiv:2401.00374 [cs.CV]"},{"key":"e_1_2_2_51_1","volume-title":"BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis. In European conference on computer vision.","author":"Liu Haiyang","year":"2022","unstructured":"Haiyang Liu, Zihao Zhu, Naoya Iwamoto, Yichen Peng, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng. 2022d. BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesis. In European conference on computer vision."},{"key":"e_1_2_2_52_1","unstructured":"Xian Liu Qianyi Wu Hang Zhou Yuanqi Du Wayne Wu Dahua Lin and Ziwei Liu. 2022b. Audio-Driven Co-Speech Gesture Video Generation. In Advances in Neural Information Processing Systems Alice H. Oh Alekh Agarwal Danielle Belgrave and Kyunghyun Cho (Eds.)."},{"key":"e_1_2_2_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52688.2022.01021"},{"key":"e_1_2_2_54_1","volume-title":"Co-Speech Gesture Synthesis using Discrete Gesture Token Learning. arXiv preprint arXiv:2303.12822","author":"Lu Shuhong","year":"2023","unstructured":"Shuhong Lu, Youngwoo Yoon, and Andrew Feng. 2023. Co-Speech Gesture Synthesis using Discrete Gesture Token Learning. arXiv preprint arXiv:2303.12822 (2023)."},{"key":"e_1_2_2_55_1","doi-asserted-by":"publisher","DOI":"10.25080\/Majora-7b98e3ed-003"},{"key":"e_1_2_2_56_1","volume-title":"Hand and Mind: What Gestures Reveal about Thought","author":"McNeill David","unstructured":"David McNeill. 1992. Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press."},{"key":"e_1_2_2_57_1","volume-title":"Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis. arXiv preprint arXiv:2306.09417","author":"Mehta Shivam","year":"2023","unstructured":"Shivam Mehta, Siyang Wang, Simon Alexanderson, Jonas Beskow, \u00c9va Sz\u00e9kely, and Gustav Eje Henter. 2023. Diff-TTSG: Denoising probabilistic integrated speech and gesture synthesis. arXiv preprint arXiv:2306.09417 (2023)."},{"key":"e_1_2_2_58_1","volume-title":"Bodytalk: A World Guide to Gestures. https:\/\/api.semanticscholar.org\/CorpusID:193353377","author":"Morris Desmond","year":"1994","unstructured":"Desmond Morris. 1994. Bodytalk: A World Guide to Gestures. https:\/\/api.semanticscholar.org\/CorpusID:193353377"},{"key":"e_1_2_2_59_1","unstructured":"Murf.AI. 2023. Murf.AI: An Online Text-to-Speech Tool. Accessed: 2023-12-15."},{"key":"e_1_2_2_60_1","doi-asserted-by":"publisher","DOI":"10.1145\/1330511.1330516"},{"key":"e_1_2_2_61_1","unstructured":"Evonne Ng Javier Romero Timur Bagautdinov Shaojie Bai Trevor Darrell Angjoo Kanazawa and Alexander Richard. 2024. From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations. In ArXiv."},{"key":"e_1_2_2_62_1","doi-asserted-by":"publisher","DOI":"10.48550\/ARXIV.2301.05339"},{"key":"e_1_2_2_63_1","unstructured":"OpenAI. 2022. ChatGPT: Optimizing Language Models for Dialogue. Accessed: 2023-05-03."},{"key":"e_1_2_2_64_1","unstructured":"OpenAI. 2023a. GPT-4 System Card. https:\/\/openai.com\/research\/gpt-4 Accessed: 2023-12-15."},{"key":"e_1_2_2_65_1","unstructured":"OpenAI. 2023b. GPT-4 Vision System Card. https:\/\/openai.com\/research\/gpt-4v-system-card Accessed: 2023-12-15."},{"key":"e_1_2_2_66_1","unstructured":"Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell Peter Welinder Paul Christiano Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv:2203.02155 [cs.CL]"},{"key":"e_1_2_2_67_1","doi-asserted-by":"publisher","DOI":"10.1145\/3592456"},{"key":"e_1_2_2_68_1","volume-title":"EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation. arXiv preprint arXiv:2305.18891","author":"Qi Xingqun","year":"2023","unstructured":"Xingqun Qi, Chen Liu, Lincheng Li, Jie Hou, Haoran Xin, and Xin Yu. 2023a. EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generation. arXiv preprint arXiv:2305.18891 (2023)."},{"key":"e_1_2_2_69_1","unstructured":"Xingqun Qi Jiahao Pan Peng Li Ruibin Yuan Xiaowei Chi Mengfei Li Wenhan Luo Wei Xue Shanghang Zhang Qifeng Liu et al. 2023b. Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation. arXiv preprint arXiv:2311.17532 (2023)."},{"key":"e_1_2_2_70_1","unstructured":"Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever et al. 2018. Improving language understanding by generative pre-training. (2018)."},{"key":"e_1_2_2_71_1","unstructured":"Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever et al. 2019. Language models are unsupervised multitask learners. OpenAI blog 1 8 (2019) 9."},{"key":"e_1_2_2_72_1","unstructured":"Rafael Rafailov Archit Sharma Eric Mitchell Stefano Ermon Christopher D. Manning and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290 [cs.LG]"},{"key":"e_1_2_2_73_1","first-page":"1","article-title":"Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer","volume":"21","author":"Raffel Colin","year":"2020","unstructured":"Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1--67. http:\/\/jmlr.org\/papers\/v21\/20-074.html","journal-title":"Journal of Machine Learning Research"},{"key":"e_1_2_2_74_1","volume-title":"In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083","author":"Ram Ori","year":"2023","unstructured":"Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083 (2023)."},{"key":"e_1_2_2_75_1","unstructured":"Yi Tay Vinh Q. Tran Mostafa Dehghani Jianmo Ni Dara Bahri Harsh Mehta Zhen Qin Kai Hui Zhe Zhao Jai Gupta Tal Schuster William W. Cohen and Donald Metzler. 2022. Transformer Memory as a Differentiable Search Index. arXiv:2202.06991 [cs.CL]"},{"key":"e_1_2_2_76_1","volume-title":"TED: Ideas change everything. Accessed: 2023-12-15.","author":"Talks TED","year":"2023","unstructured":"TED Talks. 2023. TED: Ideas change everything. Accessed: 2023-12-15."},{"key":"e_1_2_2_77_1","unstructured":"Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux Timoth\u00e9e Lacroix Baptiste Rozi\u00e8re Naman Goyal Eric Hambro Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL]"},{"key":"e_1_2_2_78_1","doi-asserted-by":"publisher","DOI":"10.5555\/3295222.3295378"},{"key":"e_1_2_2_79_1","volume-title":"\u0141 ukasz Kaiser, and Illia Polosukhin","author":"Vaswani Ashish","year":"2017","unstructured":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \u0141 ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc."},{"key":"e_1_2_2_80_1","volume-title":"Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents. 1--8.","author":"Vo\u00df Hendric","year":"2023","unstructured":"Hendric Vo\u00df and Stefan Kopp. 2023. Augmented Co-Speech Gesture Generation: Including Form and Meaning Features to Guide Learning-Based Gesture Synthesis. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents. 1--8."},{"key":"e_1_2_2_81_1","unstructured":"Melissa Wagner and Nancy Leonard Armstrong. 2003. Field Guide to Gestures: How to Identify and Interpret Virtually Every Gesture Known to Man. https:\/\/api.semanticscholar.org\/CorpusID:141961447"},{"key":"e_1_2_2_82_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.specom.2013.09.008"},{"key":"e_1_2_2_83_1","volume-title":"Weiwei Deng, Qi Zhang, and Mao Yang.","author":"Wang Yujing","year":"2023","unstructured":"Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Allen Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2023a. A Neural Corpus Indexer for Document Retrieval. arXiv:2206.02743 [cs.IR]"},{"key":"e_1_2_2_84_1","doi-asserted-by":"crossref","unstructured":"Yizhong Wang Yeganeh Kordi Swaroop Mishra Alisa Liu Noah A. Smith Daniel Khashabi and Hannaneh Hajishirzi. 2023b. Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv:2212.10560 [cs.CL]","DOI":"10.18653\/v1\/2023.acl-long.754"},{"key":"e_1_2_2_85_1","volume-title":"GESTUNO: International sign language of the deaf, langage gestuel international Des sourds","author":"World Federation of the Deaf.","year":"1975","unstructured":"World Federation of the Deaf. 1975. GESTUNO: International sign language of the deaf, langage gestuel international Des sourds. British Deaf Association [for] the World Federation of the Deaf, Carlise."},{"key":"e_1_2_2_86_1","volume-title":"Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control. arXiv preprint arXiv:2312.15900","author":"Xu Zunnan","year":"2023","unstructured":"Zunnan Xu, Yachao Zhang, Sicheng Yang, Ronghui Li, and Xiu Li. 2023. Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded Conditional Control. arXiv preprint arXiv:2312.15900 (2023)."},{"key":"e_1_2_2_87_1","volume-title":"Conversational Co-Speech Gesture Generation via Modeling Dialog Intention, Emotion, and Context with Diffusion Models. arXiv preprint arXiv:2312.15567","author":"Xue Haiwei","year":"2023","unstructured":"Haiwei Xue, Sicheng Yang, Zhensong Zhang, Zhiyong Wu, Minglei Li, Zonghong Dai, and Helen Meng. 2023. Conversational Co-Speech Gesture Generation via Modeling Dialog Intention, Emotion, and Context with Diffusion Models. arXiv preprint arXiv:2312.15567 (2023)."},{"key":"e_1_2_2_88_1","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2023\/650"},{"key":"e_1_2_2_89_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.00230"},{"key":"e_1_2_2_90_1","doi-asserted-by":"crossref","unstructured":"Sicheng Yang Zunnan Xu Haiwei Xue Yongkang Cheng Shaoli Huang Mingming Gong and Zhiyong Wu. 2024. Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP).","DOI":"10.1109\/ICASSP48485.2024.10447978"},{"key":"e_1_2_2_91_1","volume-title":"MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations. arXiv preprint arXiv:2310.10198","author":"Yao Heyuan","year":"2023","unstructured":"Heyuan Yao, Zhenhua Song, Yuyang Zhou, Tenglong Ao, Baoquan Chen, and Libin Liu. 2023. MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations. arXiv preprint arXiv:2310.10198 (2023)."},{"key":"e_1_2_2_92_1","doi-asserted-by":"publisher","DOI":"10.1109\/IROS47612.2022.9981117"},{"key":"e_1_2_2_93_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-20065-6_41"},{"key":"e_1_2_2_94_1","volume-title":"Black","author":"Yi Hongwei","year":"2022","unstructured":"Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J. Black. 2022. Generating Holistic 3D Human Motion from Speech."},{"key":"e_1_2_2_95_1","volume-title":"EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model. arXiv preprint arXiv:2306.11496","author":"Yin Lianying","year":"2023","unstructured":"Lianying Yin, Yijun Wang, Tianyu He, Jinming Liu, Wei Zhao, Bohan Li, Xin Jin, and Jianxin Lin. 2023. EMoG: Synthesizing Emotive Co-speech 3D Gesture with Diffusion Model. arXiv preprint arXiv:2306.11496 (2023)."},{"key":"e_1_2_2_96_1","doi-asserted-by":"publisher","DOI":"10.1145\/3414685.3417838"},{"key":"e_1_2_2_97_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICRA.2019.8793720"},{"key":"e_1_2_2_98_1","doi-asserted-by":"publisher","DOI":"10.1145\/3536221.3558058"},{"key":"e_1_2_2_99_1","doi-asserted-by":"publisher","DOI":"10.1109\/TASLP.2021.3129994"},{"key":"e_1_2_2_100_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-27077-2_18"},{"key":"e_1_2_2_101_1","unstructured":"Fan Zhang Naye Ji Fuxing Gao Siyuan Zhao Zhaohan Wang and Shunman Li. 2023b. Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model. arXiv:2308.05995 [cs.SD]"},{"key":"e_1_2_2_102_1","volume-title":"SpeechAct: Towards Generating Whole-body Motion from Speech. arXiv preprint arXiv:2311.17425","author":"Zhang Jinsong","year":"2023","unstructured":"Jinsong Zhang, Minjie Zhu, Yuxiang Zhang, Yebin Liu, and Kun Li. 2023d. SpeechAct: Towards Generating Whole-body Motion from Speech. arXiv preprint arXiv:2311.17425 (2023)."},{"key":"e_1_2_2_103_1","unstructured":"Xinghua Zhang Bowen Yu Haiyang Yu Yangyu Lv Tingwen Liu Fei Huang Hongbo Xu and Yongbin Li. 2023c. Wider and Deeper LLM Networks are Fairer LLM Evaluators. arXiv:2308.01862 [cs.CL]"},{"key":"e_1_2_2_104_1","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.01902"},{"key":"e_1_2_2_105_1","doi-asserted-by":"publisher","DOI":"10.1145\/3536221.3558063"},{"key":"e_1_2_2_106_1","unstructured":"Yujia Zhou Jing Yao Zhicheng Dou Ledell Wu and Ji-Rong Wen. 2022b. DynamicRetriever: A Pre-training Model-based IR System with Neither Sparse nor Dense Index. arXiv:2203.00537 [cs.IR]"},{"key":"e_1_2_2_107_1","volume-title":"Multi-Part Human Motion Synthesis. arXiv preprint arXiv:2311.16471","author":"Zhou Zixiang","year":"2023","unstructured":"Zixiang Zhou, Yu Wan, and Baoyuan Wang. 2023. A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis. arXiv preprint arXiv:2311.16471 (2023)."},{"key":"e_1_2_2_108_1","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52729.2023.01016"},{"key":"e_1_2_2_109_1","unstructured":"Yutao Zhu Huaying Yuan Shuting Wang Jiongnan Liu Wenhan Liu Chenlong Deng Zhicheng Dou and Ji-Rong Wen. 2023b. Large Language Models for Information Retrieval: A Survey. arXiv:2308.07107 [cs.CL]"},{"key":"e_1_2_2_110_1","doi-asserted-by":"crossref","unstructured":"Noah Ziems Wenhao Yu Zhihan Zhang and Meng Jiang. 2023. Large Language Models are Built-in Autoregressive Search Engines. arXiv:2305.09612 [cs.CL]","DOI":"10.18653\/v1\/2023.findings-acl.167"}],"container-title":["ACM Transactions on Graphics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658134","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3658134","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,19]],"date-time":"2025-06-19T00:05:54Z","timestamp":1750291554000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3658134"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,7,19]]},"references-count":110,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,7,19]]}},"alternative-id":["10.1145\/3658134"],"URL":"https:\/\/doi.org\/10.1145\/3658134","relation":{},"ISSN":["0730-0301","1557-7368"],"issn-type":[{"value":"0730-0301","type":"print"},{"value":"1557-7368","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,7,19]]},"assertion":[{"value":"2024-07-19","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}