Fei Haolin, Xue Tao, He Yiyang, Lin Sheng, Du Guanglong, Guo Yao, Wang Ziwei
School of Engineering, Lancaster University, Lancaster, United Kingdom.
Department of Automation, Tsinghua University, Beijing, China.
Front Robot AI. 2025 Jul 17;12:1621033. doi: 10.3389/frobt.2025.1621033. eCollection 2025.
Bimanual teleoperation imposes cognitive and coordination demands on a single human operator tasked with simultaneously controlling two robotic arms. Although assigning each arm to a separate operator can distribute workload, it often leads to ambiguities in decision authority and degrades overall efficiency. To overcome these challenges, we propose a novel bimanual teleoperation large language model assistant (BTLA) framework, an intelligent co-pilot that augments a single operator's motor control capabilities. In particular, BTLA enables operators to directly control one robotic arm through conventional teleoperation while directing a second assistive arm via simple voice commands, and therefore commanding two robotic arms simultaneously. By integrating the GPT-3.5-turbo model, BTLA interprets contextual voice instructions and autonomously selects among six predefined manipulation skills, including real-time mirroring, trajectory following, and autonomous object grasping. Experimental evaluations in bimanual object manipulation tasks demonstrate that BTLA increased task coverage by 76.1 and success rate by 240.8 relative to solo teleoperation, and outperformed dyadic control with a 19.4 gain in coverage and a 69.9 gain in success. Furthermore, NASA Task Load Index (NASA-TLX) assessments revealed a 38-52 reduction in operator mental workload, and 85 of participants rated the voice-based interaction as "natural" and "highly effective."
双手遥操作对负责同时控制两个机器人手臂的单个操作人员提出了认知和协调方面的要求。虽然将每个手臂分配给单独的操作人员可以分散工作量，但这往往会导致决策权限不明确，并降低整体效率。为了克服这些挑战，我们提出了一种新颖的双手遥操作大语言模型助手（BTLA）框架，这是一种智能副驾驶，可增强单个操作人员的运动控制能力。具体而言，BTLA使操作人员能够通过传统遥操作直接控制一个机器人手臂，同时通过简单的语音命令指挥第二个辅助手臂，从而同时指挥两个机器人手臂。通过集成GPT-3.5-turbo模型，BTLA解释上下文语音指令，并在六种预定义的操作技能中自主选择，包括实时镜像、轨迹跟踪和自主物体抓取。在双手物体操作任务中的实验评估表明，相对于单人遥操作，BTLA将任务覆盖率提高了76.1%，成功率提高了240.8%，并且在覆盖率方面比二元控制提高了19.4%，在成功率方面提高了69.9%。此外，美国国家航空航天局任务负荷指数（NASA-TLX）评估显示，操作人员的心理工作量减少了38%-52%，85%的参与者将基于语音的交互评为“自然”和“高效”。
