整合大型語言模型至機器人感知與組裝系統

本研究探討將大型語言模型，特別是多模態語言模型，整合至機器人感知系統的方法，藉由結合影像與語言輸入，實現情境理解與自主決策。透過將視覺編碼器處理的影像資料與自然語言輸入結合，系統進行情境推理和決策，並透過多種策略量化感知結果。此方法已應用於機器人螺絲緊固組裝任務，採用Gemma 3 4B模型整合視覺、力覺感知和控制模組，建構智慧組裝系統。實驗結果顯示，將多模態感知資訊與語言模型的推理能力相結合，可以使人工智慧有效地評估任務成功率並提供可信度評估，為智慧製造中的精密組裝提供了一種新穎的解決方案。

This study explores the integration of Large Language Model (LLM), particularly Multimodal Language Model (MLLM), into robotic perception systems to enable deep environmental understanding and autonomous decision-making. By combining image data processed by visual encoders with natural language input, the system performs situational reasoning and decision-making and quantifies perception results through various strategies. The proposed approach is applied to a robotic screw-fastening assembly task, using the Gemma 3 4B multimodal language model in conjunction with visual recognition, force sensing, and control modules to establish a complete intelligent assembly system. Experimental results show that integrating multimodal sensory information with the reasoning capabilities of the language model allows the AI to effectively assess task success and provide confidence estimations, offering a novel solution for precision assembly in smart manufacturing.