Papers for

conversational ai teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Evolving programs improves agent memory accuracy and efficiency

Discovering Effective Agent Memory Designs through Program Evolution

Abstract: Program evolution provides a promising approach to discover agent memory designs and study the choices underlying their effectiveness. We introduce MAPE, a framework that evolves whole memory programs from minimal seeds through a standardized lifecycle interface and an iterative diagnosis-proposal loop. Experiments on LoCoMo and LongMemEval yield an absolute accuracy gain of up to 12% over manually designed baselines at 35% of the serving cost, with further evaluations supporting transfer across serving backbones and benchmarks. Empirical analyses and ablations identify and validate two recurring design patterns: retaining raw conversations as retrievable evidence preserves details omitted from extracted memories, while composing retrieved evidence according to query intent helps the answering model use it effectively. Transplanting the composition module into five existing memory systems improves accuracy across all five. Beyond these patterns, the search objective shapes where computation goes: optimizing serving cost alongside accuracy shifts computation from LLM-based memory construction to selective processing at query time while retaining competitive accuracy. Together, these findings offer empirical lessons for agent memory design. Code is available at https://github.com/ash-ding/MAPE.

Thu 12 NovComputation and LanguageArtificial Intelligence
The gist
Remembering details well is important for AI agents, but designing how they store and use memory can be tricky. The authors created a system called MAPE that automatically evolves memory programs to improve how agents remember and answer questions. This method found better memory designs that keep raw conversations and organize evidence smartly, which made AI answers more accurate and cheaper to get. Their findings also helped improve existing memory systems by adding better ways to combine information.
Open → 2611.17068v1

Outcome-Mass alignment improves multi-turn language model agent learning

Outcome-Mass Alignment: Controlling Length-Dependent Trajectory Influence in Multi-Turn LLM Agent Reinforcement Learning

Abstract: To keep long interaction contexts tractable, stepwise group-relative training for multi-turn LLM agents materializes each source trajectory into multiple response rows and valid tokens. This expansion creates an outer-accounting problem distinct from within-trajectory credit assignment: row-based outcome statistics can count one episode outcome multiple times, while token-level reduction can assign unequal aggregate objective mass across trajectories. We term these effects outcome-statistics multiplicity and objective-mass imbalance. We introduce Outcome-Mass Alignment (OMA), a two-component method that makes both accounting choices explicit. Outcome Alignment computes outcome statistics over unique source trajectories, while Mass Alignment controls aggregate objective mass after within-trajectory aggregation of optimizer-native token objectives. Neither redesigns the base optimizer's credit assignment, importance ratios, or clipping. Across matched ALFWorld GRPO comparisons at two model scales, where failed trajectories typically expand much more than successful ones, OMA improves matched-horizon endpoint success by 7.0-9.4 percentage points. Ordinary WebShop exhibits weak or reversed expansion-outcome coupling and correspondingly mixed effects, whereas in a success-oriented WebShop setting, WebShop-SOL, OMA improves success nAUC from 0.399 to 0.511. Component ablations and GiGPO and GSPO comparisons further support component separability and cross-optimizer compatibility. These results show that trajectory expansion always defines an implicit weighting rule, but its practical effect depends on how expansion couples to outcomes and optimizer-native updates.

Thu 12 NovComputation and Language
The gist
When teaching AI agents that hold long conversations or perform multiple steps, it's tricky to fairly measure how their actions lead to good or bad results. The authors highlight problems that happen when counting successes multiple times or giving unequal importance to different action sequences. They present a technique called Outcome-Mass Alignment (OMA) that fixes these counting issues without changing the basic learning process. OMA led to better success rates in tests with simulated environments where longer failure paths expanded more than successes.
Open → 2611.16816v1

MaxIM improves memory summaries for AI with better reward signals

MaxIM: Maximally Informative Incremental Summarization via Reinforcement Learning

Abstract: AI systems must often maintain bounded memory of unbounded interaction histories. However, optimizing the memory representation, and its updates, for downstream task utility is difficult for standard reinforcement learning (RL) due to severe credit assignment bottlenecks induced by sparse, delayed rewards. We introduce MaxIM (Maximally Informative Incremental Memory), a framework that formulates incremental summarization as a capacity-constrained, information-theoretic reward-shaping problem. Grounded in value suboptimality bounds, we show that bounding sequential information loss motivates maximizing the predictive log-likelihood of future, task-relevant targets, a variational lower bound on predictive sufficiency. We introduce a recursive consistency condition to ensure summaries are grounded by maximizing mutual information (MI) with the observed history. We mitigate credit assignment issues with a parameterized asymmetric critic that computes step-wise variational MI bounds for dense reward shaping. Empirical evaluation on long-horizon scientific reading and multi-turn dialogue benchmarks shows that MaxIM outperforms RL, supervised fine-tuning, and prompt-focused baselines on factual grounding, comprehensive autorater metrics, and downstream task utility across strict token budgets and extended horizons, without the need for expensive human preference labels.

Mon 9 NovArtificial IntelligenceMachine Learning
The gist
AI systems often need to remember important facts from long interactions but struggle because they get feedback only rarely and late. The authors developed MaxIM, a method that teaches AI to make short memory summaries that keep the most useful information for future tasks. It does this by using an information-based way to score memory steps more often, helping the AI learn which details to keep. Tests on reading scientific papers and dialogue tasks show MaxIM keeps facts better and helps AI perform tasks more reliably without needing lots of human feedback.
Open → 2611.12825v1

Relational structure helps language models connect meanings across languages

Leaving the Cave by Breaking the Symmetry: Identifiable Structure Can Enable Cross-Lingual Generalization

Abstract: Why do multilingual language models generalize across languages even without shared vocabulary? The Platonic Representation Hypothesis posits that models trained on different data, objectives, and modalities converge toward a shared statistical model of reality. Extending this intuition to representations inside a single multilingual model, we propose the Relational Identifiability Hypothesis: relations in a shared world can promote entity alignment by ruling out alternative cross-lingual correspondences, making corresponding entities more likely to become directly readable without an additional transformation. We test this hypothesis in a controlled bilingual Transformer whose two token-disjoint languages describe the same modular-arithmetic world. We find that the languages can develop a shared answer subspace without aligning corresponding numbers. Cross-lingual alignment emerges selectively when the available relations constrain number correspondences: subtraction makes zero task-level identifiable, inducing selective alignment, while an order-sensitive operation produces broad alignment across numbers. These patterns predict success in zero-shot cross-lingual transfer. Together, our results show that relational structure in a shared world can promote entity alignment and enable transfer.

Wed 28 OctComputation and Language
The gist
Languages can be very different, so it’s hard for language models to understand things in one language by learning from another without shared words. The authors show that when two languages describe the same relationships in the world, like math operations, the model can align meanings across languages better. This happens because the relationships reduce the number of ways to match things, making some connections clearer. Their experiments with a simple bilingual Transformer show this alignment depends on the type of relationships, which also predicts how well the model transfers knowledge.
Open → 2610.38509v1

Lookahead self-distillation improves training of language models for reasoning

Teach Me After You Learn It: Lookahead Self-Distillation for RLVR

Abstract: On-policy self-distillation (OPSD) has emerged as a popular training paradigm for LLMs, providing dense, token-level supervision by aligning a model's own distribution with the distribution induced by privileged information. However, existing OPSD methods derive supervision solely from privileged information of the same instance, leaving inter-instance supervision largely unexplored. We propose Lookahead Self-Distillation (LEAD), which introduces inter-instance distillation by allowing the teacher to first learn from other on-policy rollouts before teaching the student. LEAD constructs lookahead teachers through virtual RL updates on disjoint subsets of on-policy rollouts, and uses each teacher to provide cross-group token-level supervision for the other subset. To reconcile the complementary yet potentially conflicting signals from OPSD and RLVR, we introduce harmonious modulation, which adaptively modulates the GRPO advantage based on the agreement between the two signals. Extensive experiments on scientific reasoning and interactive-agent benchmarks show that LEAD achieves the best in-domain average performance across model scales while maintaining out-of-domain generalization comparable to GRPO.

Tue 13 OctMachine Learning
The gist
Teaching large language models to reason well is hard because they usually learn only from their own mistakes on the same example. The authors propose a new method called LEAD, where the model learns by looking ahead and learning from other related examples too. This method combines two ways of teaching the model so it gets better signals without confusion. Tests show that models trained this way get better at reasoning tasks within their training area without losing skill when facing new problems.
Open → 2610.18256v1

Vision language models improve understanding of negation questions

Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models

Abstract: Despite the strong performance of Vision-Language Models (VLMs) on a wide range of visual question answering (VQA) tasks, these models consistently struggle to understand negation and produce incorrect answers when questions involve negated clauses. To address this limitation, we propose Skeleton-and-Strategy Prompting (\textbf{SSP}), a training-free, in-context learning method that improves VLM negation understanding capabilities without any parameter updates. Given a negation question, our method first abstracts the underlying question structure into a skeleton, retrieves a small set of same-skeleton questions from a lightweight question pool, then prompts the VLM to analyze their shared negation pattern and synthesize a single-sentence answering strategy. The skeleton and strategy are prepended to the test sample to guide the model correctly tackle the negation problems. Experiments on multiple negation VQA benchmarks show that SSP achieves state-of-the-art performance on negation-focused VQA tasks while remaining computationally efficient.

Thu 1 OctComputer Vision and Pattern Recognition
The gist
Vision-language models are good at answering many visual questions but have trouble with questions that involve negation, like 'What is not in the picture?'. The authors introduce a new method called Skeleton-and-Strategy Prompting (SSP) that helps these models understand negation without any extra training. SSP works by breaking down the question into a basic structure, finding similar questions, and then teaching the model a simple way to answer negation questions. Tests show this method improves the model’s accuracy on negation tasks efficiently.
Open → 2610.01180v1

New method improves language model learning from verbal feedback

Inductive Feedback for Mixed-Policy Distillation

Abstract: Verbal feedback can identify errors and prescribe corrections, providing rich supervision for language-model post-training even when reliable programmatic verifiers are unavailable. Such feedback, often generated by a capable model, can be used to condition the teacher in on-policy distillation, which trains the student to match the teacher's predictions on student-generated rollouts. However, this approach can transfer teacher preferences that the feedback did not motivate, while leaving much of the feedback's guidance unused. We find that both problems come from the standard on-policy distillation objective, specifically the divergence it minimizes and the distribution it uses as its target. Our proposed method addresses both limitations. First, to isolate the information conveyed by the feedback from the teacher's inherent preferences, we treat verbal feedback as evidence for or against the hypothesis that a particular token comes next at a given prefix. We then adopt a probabilistic confirmation framework which uniquely determines an ordering over the vocabulary based on the teacher's predictions before and after it receives feedback. Using a confirmation score consistent with this ordering, we construct a target distribution within a trust region of the student. Second, to learn from guidance that student rollouts can leave unused, we derive a simple shared-rollout estimator of a symmetric divergence between the student and target distributions over rollouts, reusing student and feedback-conditioned teacher rollouts in both directions through importance weighting. Empirical evaluations show that our method outperforms the common on-policy distillation recipe and a recent contrastive variant on knowledge-based and agentic benchmarks.

Mon 28 SeptMachine Learning
The gist
Sometimes language models can be taught better by using verbal feedback that points out mistakes and suggests fixes. The authors found that current ways of using this feedback can accidentally transfer the teacher model's own biases or ignore much of the feedback. They developed a new approach that treats feedback as clues for what the next word should be and carefully updates the student model’s learning. Their method also makes better use of the feedback even when the student model doesn’t immediately apply it. Tests show this approach works better on tasks involving knowledge and agent actions.
Open → 2609.35390v1

LLM agents struggle to plan and update personalized tool tasks over time

PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents

Abstract: Large language model (LLM) agents are evolving from tool-calling systems that execute isolated instructions into task-oriented agents that pursue user goals through sustained, multi-step interactions. However, existing benchmarks for personalized tool use largely assess isolated calls or reactive execution, leaving unclear whether agents can formulate, execute, and revise an explicit plan while preserving user preferences throughout long-term interaction. To address this gap, we introduce \textbf{PDEU-Bench} (\textbf{P}ersonalized plan \textbf{D}efinition, plan \textbf{E}xecution, and plan \textbf{U}pdate \textbf{Bench}mark), a benchmark for evaluating the complete planning lifecycle of personalized tool-using agents. PDEU-Bench comprises 214 long-horizon interaction tasks spanning 12 everyday domains and 94 tools, with stage-specific assessments of preference adherence and plan quality. Extensive evaluations of 15 representative open-source and closed-source LLMs reveal a pronounced gap between local tool execution and dynamic planning: LLMs can often instantiate preferences in individual calls, yet struggle to construct coherent plan definition and plan update. We further evaluate mainstream personalization and memory-augmentation methods. Although these methods improve particular stages, none of the evaluated methods reliably propagates user preferences throughout the complete lifecycle, and their gains frequently fail to transfer to subsequent execution. Fine-grained error analysis further reveals that preference omissions and conflicts persist throughout the planning lifecycle, highlighting the need for future research to parameterize LLMs with preference-aware information retrieval and memory capabilities. We provide the relevant code and data in the appendix to support future research.

Mon 28 SeptArtificial Intelligence
The gist
People want AI agents that can carry out multi-step tasks while remembering what users prefer over time. The authors created a new test called PDEU-Bench to check how well language model agents can make, follow, and adjust plans while respecting personal preferences. They found that while models handle individual tool calls okay, they have trouble with overall planning and updating plans. Current personalization methods help somewhat but don’t fully solve these problems yet.
Open → 2609.34930v1

Method improves diversity of AI text generation without trade offs

Improving the Diversity of LLM Outputs without a Trade-off

Abstract: We propose DAST (Diversifying Arithmetic Sampling with TokenTour), a method that increases the diversity of LLM outputs without any change to the marginal distribution and with negligible generation-time overhead (a few microseconds). We observe that token IDs are often arranged in a meaningless order and reassign them so that tokens with similar meanings appear consecutively. This can be done in advance in a few hundred seconds per model, and the resulting order can be reused for all subsequent generations. By combining this order with arithmetic sampling (or quasi-Monte Carlo methods), we make similar tokens less likely to be generated across runs while preserving the distribution. Our method not only produces qualitatively good ideas but also significantly improves performance on the downstream task of ProtoQA.

Sun 27 SeptComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Language models sometimes produce similar or repetitive responses, limiting the variety of ideas they generate. The authors propose a method called DAST that rearranges how tokens (words or pieces of words) are ordered internally so that similar meanings appear together. This change helps make similar tokens less likely to be picked repeatedly, increasing output diversity without slowing down generation or changing what the model thinks is likely overall. Their approach improves results on a question-answering task, showing it can produce a wider range of good answers.
Open → 2609.33038v1

Language models improve counting by thinking and revisiting evidence

Counting on Thinking: Tracing Evidence Integration in Language Models

Abstract: Finite computational resources force a tradeoff between automatic System 1 processes and costly System 2 thinking. Large language models (LLMs) can spend extra computation on hard problems, yet direct answers struggle even with counting, an elementary operation humans and animals perform automatically. We ask why this requires thinking in LLMs. Evidence integration has long been used in psychology and neuroscience to probe decision-making. Our evidence-integration task presents one letter per conversational turn and asks which of two target letters appeared more often. A running count difference solves the task optimally by weighting every letter equally; tokens at each turn could represent and update this difference. Direct responses instead weighted evidence unevenly, with strong recency effects, and assigned less probability to the correct answer as difficulty increased. Thinking improved performance and made integration weights nearly uniform, yet final-query attention remained concentrated on the sequence ends in both modes. Reasoning trajectories showed models revisiting input, recounting letters, and checking intermediate counts that informed the answer, suggesting that thinking constructs the accumulated count that direct responses lack rather than reading out one already formed. Reasoning-token costs grew with the number of letters far more than with coherence. Outcome feedback did not bring this computation into direct responses: under in-context reinforcement learning (ICRL), performance deteriorated over repeated games and recency effects strengthened, yet models grew more confident. Humans and animals amortize such computations into automatic processes, whereas current LLMs still pay for them with thinking on every trial. Which operations learning can make directly available remains central to how future models allocate computation.

Sat 26 SeptMachine Learning
The gist
Counting which letter appears more in a sequence is easy for humans but hard for large language models (LLMs). The authors studied why LLMs struggle by giving them tasks where they must keep track of letters one at a time. They found that without extra 'thinking' steps, LLMs guess based more on recent letters and perform worse as tasks get harder. When asked to think longer, the models do better by reviewing and updating counts more evenly. However, unlike humans, LLMs still have to spend effort on counting each time instead of doing it automatically.
Open → 2609.32932v1

Moscopt improves llm agents by mixing and choosing skills dynamically

MOSCOPT: Mixture-of-Skills Collective Optimization for LLM Agents

Abstract: Natural language prompts and skills serve as the strategic backbone of LLM-based agents. Recent advances in prompt and skill optimization have achieved notable gains, yet all existing methods optimize a \emph{single} text template---missing the synergy among multiple complementary strategies. We propose MOSCOPT, a text-native, parameter-free algorithm that jointly optimizes a pool of $N$ skills and a gating skill $G$ that dynamically selects $K$ skills per step. To effectively optimize the skills, we build the EditAdam with internally maintained dual states. Through the three-phase interleaved updates with EditAdam, the system monotonically improves without gradient or parameter tuning. Extensive experiments and detailed ablations across 5 benchmarks and 3 target LLMs demonstrate that MOSCOPT consistently outperforms all baselines, and confirm that both the mixture-of-skills architecture with selective activation and the collective evolution with three-phase interleaving are essential to its superior performance. Code is released https://github.com/zhangzhenyu13/SummerClaw/tree/master/summerclaw/agent_trainer/algorithms/moscopt.

Sun 13 SeptArtificial IntelligenceComputation and Language
The gist
LLM agents use language prompts called skills to perform tasks, but existing approaches optimize only one skill at a time, missing benefits from combining multiple strategies. The authors propose MOSCOPT, a method that optimizes a group of skills together and learns when to activate which skills dynamically. They introduce a new optimization technique called EditAdam to improve these skills without needing parameter tuning. Experiments show MOSCOPT performs better than previous methods by using a mix of skills and updating them collectively in an organized way.
Open → 2609.14399v1