Papers for

ai assistant developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Multimodal retrieval struggles to fully satisfy complex evolving user requests

TRIAD: Multimodal Retrieval across Abstraction, Constraints, and State

Abstract: Multimodal retrieval increasingly supports agentic systems, where user requests can involve cross-level relations, combine multiple constraints, and evolve across dialogue turns. In these settings, a highly similar result may still violate the user's requirements, while retrieved content may directly inform downstream actions without human screening. We introduce TRIAD, a multimodal retrieval benchmark for evaluating how well retrieval systems satisfy these complex, evolving requests. TRIAD comprises three tracks: abstraction, testing bidirectional retrieval between visual instances and concepts at different hierarchical levels; constraints, testing the joint satisfaction of positive and negative conditions; and state, testing the integration of requirements across dialogue turns and the updating of earlier conditions when users revise their requests. The benchmark contains six subsets, each with 1,000 queries and a fixed pool of 1,000 candidates per query, evaluated using a common multi-positive mean average precision (mAP) protocol. Simpler reference tasks and hard negatives grouped by the requirements they violate help characterize specific retrieval difficulties. We evaluate 20 configurations of 17 systems and find that no system leads across all six subsets. Performance declines with hierarchical distance, and negative-only queries generally underperform positive-only queries with the same number of constraints. Across paired tasks expressing the same final intent, multi-turn retrieval yields lower mAP in 39 of 40 configuration-domain comparisons. The benefits of explicit reasoning also vary across models and tasks. These findings highlight the gap between retrieving similar content and satisfying a complete request.

Mon 9 NovComputer Vision and Pattern Recognition
The gist
When people ask computers to find things that combine images and words, the requests can be very tricky. The paper by the authors shows that current search systems have trouble fully understanding complicated demands that change over time or include both what to include and exclude. They created a new test called TRIAD to measure how well these systems do with tasks involving different levels of detail, positive and negative criteria, and changing requirements in a conversation. Their tests show no system performs great on all parts, and handling negative or changing requests remains especially difficult.
Open → 2611.12364v1

Progressive retrieval improves complex question answering from multiple web pages

PCG-RAG: Progressive Compositional Grounding for Retrieval-Augmented Generation in Complex Web Question Answering

Abstract: Complex Web question answering requires retrieval-augmented generation (RAG) systems to identify target entities from multi-clue descriptions and connect evidence scattered across long Web pages. Individual pages often support only some of the required semantic constraints, while clues discovered on one page guide subsequent searches on others. However, premature reliance on partially supported entity hypotheses can divert retrieval from the original information need. RAG systems must therefore use partial evidence to guide further searches while seeking the evidence needed to establish the intended entities. We propose PCG-RAG, a novel RAG framework based on Progressive Compositional Grounding (PCG). Progressive Evidence Acquisition directs subsequent searches toward missing evidence for candidate attributes and relations. Compositional Reasoning separates candidate-guided retrieval from evidence-based answering along a directed acyclic graph (DAG) of subquestions. Provisional names make dependent searches specific, while answering tasks use accepted results and unresolved identifying conditions, assessed against the original question. New passages connect the identifying clues to the requested relations, supporting evidence-based downstream answers. Evidence Grounding combines identifying and relational source passages with supported intermediate answers, connecting evidence across Web pages to the original conditions during final generation. On BrowseComp-Plus and WebWalkerQA, PCG-RAG improves accuracy over the leading compared baselines by 5.43 and 5.15 percentage points, respectively.

Sun 8 NovInformation Retrieval
The gist
Answering complex questions from the web is hard because clues are scattered across many long pages, and no single page usually has all the information needed. The authors propose a new method that first finds partial clues and uses them to guide further searches, gradually collecting all the needed evidence before answering. Their approach keeps track of partial answers and refines them step-by-step to avoid getting distracted by incomplete information. This method works better than previous ones, showing notable improvements on challenging web question datasets.
Open → 2611.11113v1

Interactive visual gym evaluates AI assistants in everyday tasks

Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

Abstract: Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Gemini 4 Argon, completes only 53.4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.

Thu 5 NovComputer Vision and Pattern Recognition
The gist
Helping people with physical tasks using AI needs tests where the AI and human actions can affect each other. The authors built Interactive Visual Gym, which lets AI assistants work with simulated users in virtual worlds like kitchens and homes. Their tests showed current AI models struggle to give useful advice and act quickly enough during tasks. They confirmed their test results agree well with real human opinions and showed the system can help improve AI through practice.
Open → 2611.08286v1

Scientific paper names cause confusion across research fields

"PRISM" Is Not All You Need: Cross-Topic Name Collisions and Scientific Paper Retrieval

Abstract: Scientific names increasingly connect papers from different research topics, leaving readers to recover a paper's identity from context. We study this convergence in 564,861 AI-related arXiv records. Cross-topic name collision grows 4.33-fold from a pooled 2020-2022 reference to 2026, compared with 1.12-fold within topics. A second sentence encoder reproduces the pattern, and name-group resampling quantifies its uncertainty. Fixed word-frequency bands locate the main increase in an intermediate-frequency vocabulary shared by many research areas. We then measure the ambiguity that remains after supplying research clues. With the same three-term query construction, shared-name targets retain matched identification deficits of 25.7 percentage points under title-based BM25 and 6.9 points under title-plus-abstract E5. Within fixed same-name candidates, listwise language-model ranking reaches 89.2% using the query and candidate titles. Topic-specific comparisons show that context resolves more cross-topic ambiguity. The evaluation separates candidate competition, document fields, and contextual ranking through lexical, dense, and language-model baselines. Source-linked cases and a portable evaluation collection make the phenomenon inspectable and reusable for scientific retrieval and research assistants.

Sun 1 NovInformation RetrievalDigital Libraries
The gist
Scientific papers often use names that appear in multiple research areas, making it hard to find the right paper just by its name. The authors studied over half a million AI papers and found this name confusion is growing much faster between topics than within a single topic. They tested different search methods and found that adding context like abstracts helps reduce confusion but doesn’t eliminate it entirely. Their work provides tools and data to help improve how scientific papers are found despite these naming overlaps.
Open → 2611.01362v1

Omni large language models struggle with timely and proactive help

Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams

Abstract: Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions. We introduce \textbf{Omni-InteractScope}, a benchmark of 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups. It links input modalities, event triggers, intervention types, and user needs, covering event-timed tool invocation and query-free assistance alongside notifications, warnings, state tracking, and continuous responses. Our experimental results on the proposed benchmark reveal three critical limitations of current Omni-LLMs: (1) \textbf{Timely tool invocation remains unreliable}: all 12 evaluated models produce an on-time first response for at most 40\% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 32.0\%. (2) \textbf{Query-free reminders remain poorly covered}: all evaluated models achieve lower timely reference-target recall on query-free reminders than on explicit questions, measured on their respective target sets. Reminder recall never exceeds 23.0\%. (3) \textbf{End-to-end guidance coverage remains limited}: nine models cover none of the 26 guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls further show that tool-invocation scores alone do not establish event grounding and that reminder coverage depends on response history and frequency. These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.

Mon 26 OctArtificial IntelligenceComputer Vision and Pattern RecognitionHuman-Computer Interaction
The gist
Modern AI models that watch and listen continuously to videos and sounds are expected to help users with tasks as events happen. The authors created a new benchmark called Omni-InteractScope that measures how well these models act on events when needed, like starting tools or giving reminders without being asked. They found that current models often respond too late, struggle with unprompted reminders, and rarely provide full step-by-step help. This shows there is still much work to do before such AI assistants work reliably in real-time situations.
Open → 2610.35439v1

User representations improve AI assistant simulation fidelity and diversity

Learning User Representations to Improve Simulation Fidelity

Abstract: User simulators enable interactive training and evaluation of AI assistants at scale. Yet current simulators struggle to reproduce the diverse behaviors of real users, leading to assistants that generalize poorly to underrepresented users. We identify user representations as an underexplored bottleneck in user simulation. User representations are natural-language descriptions that condition simulators to reproduce a particular user's behavior. A representation's quality becomes apparent in hindsight, through the behavior it induces in a simulator. To evaluate and optimize user representations, we formulate a conversation reconstruction task: given an original conversation, we generate a representation, condition a simulator on it, simulate a conversation, and then measure fidelity by comparing user behavior in the original and simulated conversations. This comparison also yields hindsight feedback, which identifies where the representation falls short and how it can be improved. We introduce hindsight distillation, an online training procedure that distills feedback-conditioned improvements into the representation generator, enabling it to generate improved representations directly from the original conversation. We present UserRepGen-4B, a representation generator trained with hindsight distillation. Across 700 conversations spanning seven datasets, hindsight feedback helps improve user representations at test time, and UserRepGen-4B generates representations that enable more faithful simulation than those generated by GPT-5.5. Beyond conversation reconstruction, learned representations improve simulation fidelity for unseen tasks, better preserve the population-level diversity of real users, and better distinguish users across conversations.

Thu 15 OctComputation and Language
The gist
AI assistants are trained using simulated users, but these simulators often fail to mimic real people accurately, especially less common user types. The authors show that better descriptions of users, called user representations, help simulators act more like real users. They created a method to improve these representations by comparing simulated conversations to the original ones and using that feedback to make the descriptions better. Their improved system, UserRepGen-4B, generates user representations that lead to more realistic and diverse simulated users across many tasks.
Open → 2610.22518v1

Visual routing method greatly improves multi-step image retrieval for agents

Learning to Route in Visual Space via Multi-Step Embedding Retrieval

Abstract: LLM agents rely on retrieval tools to access external knowledge, yet visual agentic search remains severely bottlenecked by standard single-step retrievers. In current pipelines, the agent must issue text queries for every intermediate step, struggling when visual clues are difficult to describe or when the retriever fails to surface necessary intermediate evidence within its top results. We hypothesize that offloading multi-step navigation across the entire embedding space directly to the retrieval tool resolves this performance bottleneck. To study this systematically, we introduce VHOP, a flexible data generation framework and benchmark with five core difficulty levels testing both visual matching and search planning. Using this framework, we develop VHOP-Router, an end-to-end training pipeline---combining supervised fine-tuning, online imitation learning, and reinforcement learning---that transforms a standard embedding model into an autoregressive multi-step retriever. Operating directly in the visual latent space, VHOP-Router retrieves linked image chains in a single tool call without requiring the agent to formulate intermediate text queries. Experiments show VHOP-Router boosts retrieval performance from under 5\% to 76.3\%. In agentic search, it improves task success rates by 52.7\% and reduces the average token length by 61\% from 1886 to 728, whereas upgrading the agent yields only a 3.7\% gain. Compared to a strong baseline where the agent retrieves the top 50 results per step, VHOP-Router maintains superior performance while reducing in-context images by $23\times$ and cutting the cumulative API payload by $35\times$. The models also generalize robustly to unseen difficulty levels and realistic test sets. Ultimately, VHOP and VHOP-Router provide an efficient and effective solution for visual agentic search that leaves native LLM capabilities entirely intact.

Wed 30 SeptArtificial IntelligenceInformation Retrieval
The gist
Finding information in pictures step-by-step is hard when agents have to ask questions at every stage. The authors show that letting a retrieval system navigate images in multiple steps directly, without needing new text queries each time, works much better. They built a benchmark called VHOP and a new system, VHOP-Router, that retrieves image chains in one call. This approach improved success rates a lot, used much less text, and kept working well on new problems.
Open → 2609.38743v1

Stashbird cuts memory tokens for ai agents with indexed updates

Stashbird: Efficient Speaker-Indexed Memory for Conversational Agents

Abstract: AI agents require memory that preserves information across user-agent exchanges, user-to-user conversations, and group conversations with or without agent participation, while supporting updates as evidence changes or is removed. We present Stashbird, an agent memory system that links source episodes to derived memory state through explicit provenance. Stashbird organizes memory into episodic records, semantic relations, community summaries, and persisted graph state, with lifecycle operations for incremental updates and episode-level deletion. We evaluate question-answering accuracy and model-facing workload across four long-term memory benchmarks. On LoCoMo, Stashbird uses 76.4x fewer ingestion prompt tokens than Graphiti. Compared with reproduced Hindsight on the same benchmark, it uses 8.1x fewer retrieval prompt tokens, with accuracy 1.6 percentage points lower. It achieves higher accuracy than Hindsight on LongMemEval-S and GroupMemBench and comparable accuracy on EverMemBench.

Mon 28 SeptArtificial IntelligenceMachine Learning
The gist
AI assistants need to remember past conversations accurately, including who said what and when information changes. The authors created Stashbird, a system that organizes memory with clear links to original conversations and allows easy updates or deletions. This system uses much fewer resources to process and recall information than some existing methods, while maintaining similar or better accuracy. Stashbird is tested across multiple benchmarks involving long-term conversational memory.
Open → 2609.34242v1

Agent behaviors on tasks reveal different styles beyond success

AgentHabit: Characterizing Distinct Behaviors of Agents on Everyday Tasks

Abstract: Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways. Even when their answers are useful, how agents carry out these tasks may not match users' preferences and needs. For example, agents differ in whether they ask clarifying questions or search the web. We introduce HABIT, a taxonomy of 23 behavioral axes in five categories, which three authors and three LLMs derive bottom-up from 408 agent trajectories across 17 domains. On held-out tasks, HABIT distinguishes models more clearly than existing taxonomies of human values and agent actions while supporting comparably consistent annotation. Building on HABIT, we construct AgentHABIT, a benchmark that profiles each agent's behavioral tendencies from its trajectories on 86 everyday tasks. Profiling 18 models with AgentHABIT reveals a range of distinctive tendencies. For example, most GPT and Claude models state their assumptions and offer alternatives when requirements conflict, whereas Qwen and Google's models more often leave assumptions or changes to requirements unstated. These profiles remain recognizable even when built from entirely different sets of tasks, indicating that they reflect general tendencies rather than task-specific behavior. Prompting agents to adopt specific behaviors shifts some axes readily but barely changes others, while fine-tuning on another model's trajectories changes only part of a model's profile and leaves much of it intact. Overall, HABIT and AgentHABIT provide a systematic framework for characterizing how agents carry out everyday tasks beyond task success, offering insights to guide the development of agents whose behavior better fits users' needs.

Sat 26 SeptArtificial Intelligence
The gist
When AI assistants complete everyday tasks, they often do so in many different ways that may not always match what people prefer. The authors created a detailed checklist called HABIT to describe 23 types of behaviors that these AI agents show, like asking questions or searching online. They tested 18 agents on many tasks and found clear patterns in how each tends to behave. This helps us understand AI assistants better and could guide making them act more like people want.
Open → 2609.32795v1

Data artists develop stories and visuals together through sketches and prototypes

Understanding Creative Design Practices among Data Artists

Abstract: Creative and artistic data visualizations communicate stories and invite engagement, yet how designers develop their expressive forms remains poorly understood. We investigate the design process of data artists through three complementary studies: an analysis of 40 public project accounts, artifact-anchored interviews with seven experienced data artists, and a design task-based study with eight data artists. We find that stories and visual forms develop together through data exploration, reference adaptation, sketching, and prototyping. Inspiration comes from data, existing work, everyday imagery, and personal experience. Sketches help develop mappings, while prototypes with real data can reshape representations and intended stories. Designers consider multiple possibilities but typically develop one direction at a time, partly because producing alternatives is costly. Their choices balance meaning, visual appeal, readability, and feasibility. These findings inform tools connecting stories, references, and real data, including AI assistance that supports testing and revising alternatives while preserving designers' creative judgment.

Wed 23 SeptHuman-Computer Interaction
The gist
Creative data visualizations tell stories and invite people to explore information, but how artists design these visuals is not well known. The authors studied data artists through projects, interviews, and design tasks, finding that artists combine exploring data with sketching and prototyping to shape their stories and images. They get ideas from data, other artworks, everyday sights, and personal memories. Artists usually focus on one design direction at a time since making multiple options is costly, balancing meaning, beauty, clarity, and what they can do.
Open → 2609.28715v1

FlowSeal stops privacy leaks in AI assistants using outside control

Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control

Abstract: Personal AI agents built on large language models (LLMs) are increasingly given access to a user's private data and communications in order to provide personalized assistance. This access creates a persistent privacy risk: the agent must decide whether a given sensitive information should be disclosed to a particular party. Existing defenses address this by making the agent's backend LLM more privacy-preserving through stronger system prompts, training, or explicit consent-checking procedures, but this approach has a structural challenge: whenever enforcement is a judgment the LLM makes over the same conversational context an adversary controls, the enforcement mechanism and the attack surface coincide. We demonstrate this against existing defenses with three new attacks that require only ordinary agent interaction and no prompt injection: Collaborative Workspace Lure reframes an extraction attempt as collaborative work; Semantic Obfuscation Attack induces disclosure through omission rather than through anything the agent writes; and Channel Decoupling Attack splits the extraction request and the disclosure across independent channels. All three achieve substantially higher leak rates than the attacks these defenses were originally designed to withstand. Guided by this observation, we present FLOWSEAL, a defense that enforces confidentiality through a tool-level interceptor outside the LLM's context, grounded in data provenance and an information-flow-control lattice with controlled declassification. Evaluated across three benchmarks, five prompt-based baselines, and eight attacks, including a real agent executing live tool calls through MCP, FLOWSEAL reduces leak rates to near zero (e.g., 52.2% to 0.5% against Collaborative Workspace Lure) while preserving task utility, regardless of the underlying LLM backend.

Sat 12 SeptCryptography and SecurityArtificial Intelligence
The gist
Personal AI assistants that use large language models (LLMs) often need to access sensitive user information, creating risks that private data might be leaked. The paper's authors show that existing privacy protections can be bypassed by clever tricks that confuse the AI inside the same conversation. They introduce FlowSeal, a new defense that controls data flow outside the AI’s internal context, using careful tracking of information and limits on what can be shared. Tests show FlowSeal greatly reduces leaks while still allowing useful tasks to be done, no matter which LLM powers the assistant.
Open → 2609.14003v1