Elisabeth Menendez, Santiago Martinez, Carlos Balaguer, Michael Gienger, Anna Belardinelli, "SemanticScanpath: Combining Gaze and Speech for Situated Human-Robot Interaction Using LLMs", arxiv, 2025.

Large Language Models (LLMs) have substan- tially improved the conversational capabilities of social robots. Nevertheless, for an intuitive and fluent human-robot inter- action, robots should be able to ground the conversation by relating ambiguous or underspecified spoken utterances to the current physical situation and to the intents expressed non verbally by the user, for example by using referential gaze. Here we propose a representation integrating speech and gaze to enable LLMs to obtain higher situated awareness and correctly resolve ambiguous requests. Our approach relies on a text-based semantic translation of the scanpath produced by the user along with the verbal requests and demonstrates LLM’s capabilities to reason about gaze behavior, robustly ignoring spurious glances or irrelevant objects. We validate the system across multiple tasks and two scenarios, showing its generality and accuracy, and demonstrate its implementation on a robotic platform, closing the loop from request interpretation to execution.