Abstract
In social robotics, a pivotal focus is enabling robots to engage with humans in a more natural and seamless manner. The emergence of advanced large language models (LLMs) has driven significant advancements in integrating natural language understanding capabilities into social robots. This paper presents a system for speech-guided sequential planning in pick and place tasks, which are found across a range of application areas. The proposed system uses Large Language Model Meta AI (Llama3) to interpret voice commands by extracting essential details through parsing and decoding the commands into sequential actions. These actions are sent to DRL-VO, a learning-based control policy built on the Robot Operating System (ROS) that allows a robot to autonomously navigate through social spaces with static infrastructure and crowds of people. We demonstrate the effectiveness of the system in simulation experiment using Turtlebot 2 in ROS1 and Turtlebot 3 in ROS2. We conduct hardware trials using a Clearpath Robotics Jackal UGV, highlighting its potential for real-world deployment in scenarios requiring flexible and interactive robotic behaviors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
We employ the 8 billion parameter version (llama3-8b-8192 m) instead of the larger counterparts, such as the 80 billion parameter model, due to larger models exhibiting a tendency to assign specific room numbers when interpreting generic commands. For instance, if asked to navigate to “TRAIL lab,” it might erroneously label it with a hallucinated room number, like “Room 111.” Additionally, we can use the smaller model onboard the robot rather than relying on remote API services, leading to quicker response times and greater practicality for real-time applications.
References
Bellingham, J.G., Rajan, K.: Robotics in remote and hostile environments. Science 318(5853), 1098–1102 (2007)
Brants, T., Popat, A., Xu, P., Och, F.J., Dean, J.: Large language models in machine translation. In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 858–867 (2007)
Breazeal, C., Dautenhahn, K., Kanda, T.: Social Robotics. In: Siciliano, B., Khatib, O. (eds.) Springer Handbook of Robotics, pp. 1935–1972. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-32552-1_72
Chae, Y., Davidson, T.: Large language models for text classification: From zero-shot learning to fine-tuning. Open Science Foundation (2023)
Chang, M., et al.: Goat: Go to any thing. arXiv preprint arXiv:2311.06430 (2023)
Chang, Y., et al.: A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol. 15(3), 1–45 (2024)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Fridovich-Keil, D., Ratner, E., Peters, L., Dragan, A.D., Tomlin, C.J.: Efficient iterative linear-quadratic approximations for nonlinear multi-player general-sum differential games. In: 2020 IEEE international conference on robotics and automation (ICRA), pp. 1475–1481. IEEE (2020)
González, D., Pérez, J., Milanés, V., Nashashibi, F.: A review of motion planning techniques for automated vehicles. IEEE Trans. Intell. Transp. Syst. 17(4), 1135–1145 (2015)
Huang, J., Chang, K.C.C.: Towards reasoning in large language models: a survey. arXiv preprint arXiv:2212.10403 (2022)
Inc., G.: Groq. (https://groq.com/) (nd). Accessed July 2024
Kästner, L., et al.: Arena 3.0: Advancing social navigation in collaborative and highly dynamic environments. arXiv preprint arXiv:2406.00837 (2024)
Lakemeyer, W.S.G., Schulz, D., Thrun, S.: The interactive museum tour-guide robot. In: Proceedings of the 15th National Conference on Artificial Intelligence (1998)
Liang, P., et al.: Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022)
LLC, G.: Cloud speech-to-text API. (https://cloud.google.com/speech-to-text) (nd). Accessed July 2024
Meta AI: Meta Llama 3. (https://github.com/meta-llama/llama3) (2024)
Nau, D., Bansod, Y., Patra, S., Roberts, M., Li, R.: Gtpyhop: a hierarchical goal+ task planner implemented in python. HPlan 2021, 21 (2021)
Nau, D., Cao, Y., Lotem, A., Munoz-Avila, H.: The shop planning system. AI Mag. 22(3), 91–91 (2001)
Nejati, N., Langley, P., Konik, T.: Learning hierarchical task networks by observation. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 665–672 (2006)
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., Kiela, D.: Adversarial NLI: a new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599 (2019)
Rana, K., Haviland, J., Garg, S., Abou-Chakra, J., Reid, I.D., Suenderhauf, N.: Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. CoRR (2023)
Ren, A.Z., et al.: Robots that ask for help: Uncertainty alignment for large language model planners. In: 7th Annual Conference on Robot Learning (2023). https://openreview.net/forum?id=4ZK8ODNyFXx
Shah, D., Osiński, B., Levine, S., et al.: LM-NAV: robotic navigation with large pre-trained models of language, vision, and action. In: Conference on Robot Learning, pp. 492–504. PMLR (2023)
Shah, P., Fiser, M., Faust, A., Kew, J.C., Hakkani-Tur, D.: Follownet: Robot navigation by following natural language directions with deep reinforcement learning. arXiv preprint arXiv:1805.06150 (2018)
Singhal, K., et al.: Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617 (2023)
Vasquez, D., Okal, B., Arras, K.O.: Inverse reinforcement learning algorithms and features for robot navigation in crowds: an experimental comparison. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1341–1346. IEEE (2014)
Xie, Z., Dames, P.: DRL-VO: Learning to navigate through crowded dynamic scenes using velocity obstacles. IEEE Trans. Rob. 39(4), 2700–2719 (2023). https://doi.org/10.1109/TRO.2023.3257549
Zhao, W.X., et al.: A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)
Zhou, C., et al.: A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT. arXiv preprint arXiv:2302.09419 (2023)
Acknowledgment
This work was funded by NSF grant CNS-2143312. The authors would like to thank Zhanteng Xie, Jared Levin, and Alexander Derrico for their assistance with the hardware experiments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Srivastava, A.K., Dames, P. (2025). Speech-Guided Sequential Planning for Autonomous Navigation Using Large Language Model Meta AI 3 (Llama3). In: Palinko, O., et al. Social Robotics. ICSR + AI 2024. Lecture Notes in Computer Science(), vol 15562. Springer, Singapore. https://doi.org/10.1007/978-981-96-3519-1_15
Download citation
DOI: https://doi.org/10.1007/978-981-96-3519-1_15
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-96-3518-4
Online ISBN: 978-981-96-3519-1
eBook Packages: Computer ScienceComputer Science (R0)Springer Nature Proceedings Computer Science
