Summary
When multiple people talk in a shared 3D virtual space with a language AI, the AI has to figure out who is talking to it. The authors created a dataset where people spoke with ambiguous directions to test how well AI models can tell whom a message is for. They found that knowing which way a speaker faces helps AIs guess correctly almost all the time if the speaker is looking at the listener. However, if the speaker looks away from the listener, some models get confused. Also, showing the AI profiles about the people helps it answer questions, but sometimes causes it to share private info not discussed openly. So, AI assistants in virtual spaces need smart ways to use body language and profile data carefully.
What this means in practice
- •For virtual environment developers: Improve AI agents in multi-user virtual rooms to identify who is speaking to them using speaker facing direction and profile context.$Commercial implications: Enables virtual world platforms to sell more responsive AI assistants that understand user intent in shared spaces.
- •For chatbot designers: Design chatbots that manage when to use private user information based on conversation context and spatial cues to avoid oversharing.
Abstract
When several people share a 3D virtual room with an LLM agent, the agent must decide not only what to say, but whether an utterance was addressed to it and, if accessible, what profile information about the others present it may use. To study both problems, we construct LookAway, a controlled corpus of 40 sessions involving 80 distinct personas and an LLM agent (1,200 turns), including ambiguous-addressee turns in which speaker orientation agrees or conflicts with the intended addressee. Across three open-weight large language models and five conditions varying which profiles the agent sees and whether it is told where each person faces (18,000 decisions), adding speaker orientation increased addressee accuracy from 56% to 99.5% when orientation was congruent, but when the speaker faced someone other than the addressee, two of the models went by where the speaker faced on more than 85% of those turns. Warning one model that orientation could be misleading reduced this only modestly. A browser-based 3D demonstrator shows the effect live: the same sentence gets an answer when the speaker faces the agent and silence when they face the other person. Providing both personas' profiles improved responses about the person being asked about, but also increased the use of profile attributes not revealed in the shared conversation, reaching 45.3% of answers for one model. More context thus improves multi-user interaction but also leads to oversharing, so shared LLM agents need mechanisms for weighing spatial cues and controlling when user-specific information enters a response.