Papers for

customer support automation teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

EvoX improves success for multi-model AI agent tasks by 3 points

EvoX: Rethinking the Agent Harness as a Runtime Policy

Abstract: Large language model (LLM) agents increasingly interact with tools and environments, yet plausible model decisions do not always translate into successful execution due to tool-call mismatches, interrupted workflows, and unsupported completion claims. We present EvoX, a Rust-based, multi-model agent runtime guided by The Second Policy, which views the harness as an Operational Policy complementing the foundation model's Semantic Policy. EvoX combines model-facing guidance with deterministic runtime control through three complementary mechanisms: Action Realization, Execution Continuity, and Outcome Grounding. To evaluate harnesses without suppressing their native behaviors, we introduce Contrast, a controlled and auditable framework integrating explicit fairness constraints, provider-bound request verification, and evidence-based adjudication. Across five benchmarks, six foundation models, and six agent harnesses, EvoX achieves the highest numerical success rate of 56.2% over 691 common task--model pairs, exceeding the average of the other five harnesses by approximately 3.0 percentage points. Its larger advantages on GLM-5.3 (+10.7 points) and Kimi K3 (+7.9 points) highlight the model-dependent effectiveness of harness design and the potential for cross-provider adaptability. Achieved without memory, evolution, or extensions, these results establish EvoX as a competitive multi-model agent runtime and underscore the importance of harness-level operational control in realizing foundation-model capabilities.

Wed 11 NovArtificial IntelligenceSoftware Engineering
The gist
Sometimes AI agents that use language models to interact with tools make good decisions but mess up when actually doing tasks. The authors present EvoX, a system that carefully manages how these AI agents act and follow through on tasks to improve success rates. They tested EvoX on many tasks and models, and it performed better than other similar systems. This shows that controlling how agents operate is important to make AI models work well in real tasks.
Open → 2611.15006v1

Jev offers cost-effective multilingual text judging with some limits

Is Jev a Good Multilingual Judge?

Abstract: Generative large language models are widely used for multilingual text evaluation, yet whether decision models that directly output judgment probabilities can serve as multilingual judges has received limited attention. We evaluate Jev against generative LLM judges on four multilingual benchmarks with human annotations, covering up to 47 language varieties and three evaluation formats: binary decisions, pairwise comparisons, and graded ratings. At approximately $1/31$ to $1/61$ of GPT-6.1's evaluation cost, Jev is competitive on most multilingual judging tasks, but falls behind GPT-6.1 on knowledge-intensive answer verification. Further analysis shows that Jev is more likely to accept incorrect answers in low-resource languages than in English and that its probability outputs help identify errors across tasks. Based on these findings, we offer recommendations on evaluation instructions, confidence-based selection, and model routing, providing empirical guidance for using Jev as a multilingual judge.

Wed 11 NovComputation and Language
The gist
Deciding if a piece of text is good or bad in many languages is hard and usually expensive with big AI models. The authors tested a model called Jev, which gives a probability about the correctness of text, to see if it can judge well across many languages. Jev works well and costs much less than a bigger model called GPT-6.1, but it struggles when checking detailed knowledge answers, especially in less common languages. The authors also suggest ways to better use Jev by looking at its confidence numbers and choosing when to ask it to judge.
Open → 2611.14861v1

Evidence linked memory improves reliability of LLM agents

Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents

Abstract: External memory enables LLM agents to improve from experience by distilling trajectories of past tasks into concise lessons for new tasks. However, memory can also lead to memory-induced degradation, where an agent fails on tasks it would solve without memory, often because it follows a lesson that is incorrect, or misapplies or misreads a valid one. This is because lessons are inferred from trajectories labeled only with task success or failure, yet are typically provided without the trajectories they came from, leaving the agent no way to check them. To address this, we introduce WikiTrace, an evidence-linked memory system designed to let agents check each lesson against its original trajectory. Specifically, motivated by the finding that a trajectory often still holds the evidence needed to catch an erroneous lesson, WikiTrace stores each lesson as a concise insight linked to a summary of why its trajectory succeeded or failed and to the raw trajectory itself. Then, on a new task, the agent starts from all insights and, whenever one lacks details needed to act, conflicts with another, or comes from a failed task, descends through its links with terminal tools to fill in missing details or correct the insight before acting. We validate WikiTrace on five agentic benchmarks, showing that it consistently outperforms existing memory approaches and a baseline given the same insights alone, while also reducing memory-induced degradation. Our code is available at https://github.com/yukyeongleee/wikitrace

Tue 10 NovArtificial IntelligenceMultiagent Systems
The gist
Large language model (LLM) agents learn from past experiences stored in memory, but sometimes this memory can mislead them by providing incorrect or confusing lessons. The paper introduces WikiTrace, which links each learning lesson to the original detailed experience that generated it. This connection lets the agent double-check lessons and avoid mistakes when tackling new tasks. Tests show that using WikiTrace helps agents perform better and avoid errors caused by faulty memory.
Open → 2611.14320v1

Multilingual routing benchmark tests language models across 12 languages and mixes

PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard

Abstract: Typed-decision encoders answer a question about a text in one forward pass, but their behaviour outside English is rarely measured on a fixed yardstick. We release PolyRoute: 322 native-language support messages in 12 languages and 9 writing systems, 156 code-mixed or romanised messages (Hinglish, Spanglish, Arabizi), 967 optional native training rows, and a protocol for a three-way department-routing decision. Labels come from Qwen3.8-27B with a blind second pass (91% of candidates kept); an independent model, gemma4-31b, agrees on 97.2% of test items, and the largest character 5-gram similarity between any test text and any training text is 0.33. A leaderboard of 13 systems gives 61.2% for the English-root checkpoint, 77.3% for the shipped multilingual one and 96.6% for the best fine-tune; TF-IDF with logistic regression reaches 91.9%. On four withheld languages the three multilingual systems score 77%, 90% and 93%. On code-mixed text the English root (85%) is nominally ahead of the shipped multilingual checkpoint (79%; p=0.134). The split resolves differences only above about 4 points. All texts and labels are LLM-generated.

Tue 10 NovComputation and LanguageArtificial Intelligence
The gist
Answering questions from text in many languages is hard because most tests focus only on English. The authors created PolyRoute, a set of messages in 12 different languages and mixed-language texts to see how well models can decide routing tasks in these diverse settings. They compared several language models and found different accuracies depending on the language and mix of text. This benchmark helps measure how good language understanding is outside English using consistent tests.
Open → 2611.13843v1

EmoHarness improves small models understanding mixed emotional signals

EmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving

Abstract: Small multimodal models can misjudge emotion when words, voice, and visual cues conflict. We present EmoHarness, which runs such a model, the frozen student, inside a harness: the prompts, settings, and verification code around its calls. A human-written starting harness, the scaffold, numbers the student's visual observations, transcript, and prosody descriptions as citable evidence and has the student weigh candidate emotions against it. In Meta-Harness Evolving (MHE), a stronger LLM, the Builder, reads the student's errors and execution traces offline and edits the harness; static checks and validation gates decide which edits are kept, and the Builder never sees evaluation items. On 5,507 MME-Emotion items, EmoHarness raises task-macro Recognition by 7.6-13.4 points over a single video-only call for all four students and both Builders. Transcripts and prosody descriptions supply most of this boost (+5.38 to +10.71); the scaffold changes the audio-augmented call by -0.79 to +2.54, partly through extra multi-label predictions, and evolved edits add at most +0.95, although no evolved increment survives Holm correction. The harness's emotion context also transfers to IntentQA, with small gains for three of four students, none of which survives correction. EmoHarness thus boosts small multimodal students chiefly by supplying and organizing audio-derived evidence.

Mon 9 NovComputation and LanguageArtificial IntelligenceComputer Vision and Pattern Recognition
The gist
Small AI models that try to understand emotions from videos can get confused when words, tone of voice, and facial expressions don’t match. The paper introduces EmoHarness, which acts like a helper around these models by organizing and numbering different emotional clues before the model makes a decision. Then, a stronger AI called the Builder reviews mistakes made by the helper and improves it without seeing test examples. The result is that these small models recognize emotions more accurately, mainly because EmoHarness better organizes audio and speech clues.
Open → 2611.11994v1

Adaptive meta-learning improves noise robustness in retrieval augmented generation

ARMOR: Adaptive Curriculum Meta-Learning for Noise-Robust RAG Reasoning

Abstract: Retrieval-Augmented Generation (RAG) systems have demonstrated remarkable effectiveness in mitigating hallucinations by incorporating external knowledge. However, the retrieval process inevitably introduces noise, posing significant challenges to RAG robustness. Fundamentally, noise robustness is a composite competency encompassing noise identification, retrieval filtering, and robust reasoning. Unfortunately, existing methods often fail to capture this holistic nature, typically restricted to single-dimensional enhancements or hampered by noise-induced optimization interference. To address these limitations, we decompose RAG robustness into three sub-tasks: knowledge boundary perception, retrieval verification, and robust reasoning. Accordingly, we propose ARMOR, an Adaptive Curriculum Meta-Learning for Noise Robust RAG Reasoning. Inspired by the teaching process, ARMOR introduces an adaptive task scheduler as a teacher to curate training plans according to the absolute learning progress of RAG generators (Student). Distinct from GRPO, ARMOR utilizes inter-group advantage estimation during learning to effectively mitigate mode collapse caused by limited output spaces. Experiments on three complex reasoning datasets (2WikiMultiHopQA, MusiQue, and HotPotQA) indicate that ARMOR achieves synergistic improvements in retrieval noise robustness, significantly outperforming baselines in output quality.

Mon 9 NovArtificial Intelligence
The gist
Retrieval-Augmented Generation (RAG) systems try to answer questions using external information, but sometimes the extra information contains mistakes or noise. The authors say that making RAG systems better at handling this noise involves three key skills: knowing when information might be wrong, checking the retrieved facts, and reasoning well despite errors. They created ARMOR, which uses a teaching-style method to help the system learn these skills step-by-step. Tests show ARMOR helps RAG systems give better answers even with noisy information.
Open → 2611.11771v1

Rule based agent wins bargaining games without real time language model calls

Beating the Pool, Not the Game: A White-Box Equilibrium Agent for Language-Based Economic Games

Abstract: We describe sudsho, a rule-based agent for the GLEE Competition that plays bargaining, negotiation, and persuasion with zero inference-time LLM calls. Games are scored as payoff percentiles against an empirical pool of prior play on the same configuration, so pool percentiles set our demand targets while clamped equilibrium shares bound demands and acceptance. Every move is a deterministic function of the visible state (persuasion mixing aside), so alignment is verified by replaying the post-freeze move log through the policy. Across 117,280 games the agent reached an overall rating of 2056 at $0 inference cost, and the account, ranked by its best agent (a same-policy replica), placed 26th of 71 ranked accounts. A replay against three frozen snapshots of the opponent pool credits the pool-aware clamp with 274 to 330 points of mean per-game bargaining rating, an effect that holds where logged opponent decisions fix the replay.

Sat 7 NovRobotics
The gist
The paper presents sudsho, an agent that plays negotiation and persuasion games without using online language models during moves. It sets its goals based on how well previous players performed, while making sure its offers and acceptances stay within safe limits using equilibrium rules. The agent's decisions depend only on what it can see in the game, making its behavior easy to verify. Tested over many games, sudsho performed well while requiring no costly language model calls in real time.
Open → 2611.10932v1

Large language models edited with geometry aware knowledge protection

Editing Large Language Models with Geometry-Aware Regularization

Abstract: Knowledge editing has become a key technique for updating knowledge in large language models. However, existing methods neglect the geometric structure of the knowledge representation space, leading to two key limitations: they cannot distinguish which pre-trained knowledge should be preferentially preserved (core vs. non-core) nor which new edits are inherently difficult (hard vs. easy). To address this, we propose GEdit, a geometry-aware framework built on a Low-Rank Plus Diagonal (LR+D) factor model that captures the intrinsic low-rank structure of Transformer FFN hidden representations. GEdit features two complementary mechanisms. First, the low-rank regularization applies targeted protection on a low-rank core knowledge subspace. Second, the adaptive regularization dynamically adjusts the regularization strength based on each edit request's editability: relaxing constraints for hard edits to ensure success, and tightening them for easy edits to better preserve pretrained knowledge. Extensive experiments on GPT2-XL, GPT-J, and LLaMA-3-8B under large-scale editing scenarios, including both mini-batch and fully sequential editing settings with up to 10,000 continuous updates, show that GEdit consistently and significantly outperforms state-of-the-art baselines in both editing success and knowledge preservation.

Wed 4 NovComputation and LanguageArtificial Intelligence
The gist
Updating the knowledge inside large language models can be tricky because some facts are more important to keep, and some updates are harder than others. The authors introduce GEdit, a new way to edit these models that respects the underlying math structure of how knowledge is stored. This method protects the most important knowledge better and adjusts how strictly it edits based on how difficult the update is. Tests on popular large language models show GEdit works better than previous methods by both keeping old knowledge intact and successfully making new edits.
Open → 2611.06185v1

ReplayRCA measures how well models fix agent failures live and after the fact

ReplayRCA: Measuring Post-hoc Diagnosis and Live Oversight of Agent Failures by Counterfactual Replay

Abstract: When a language-model agent fails a multi-step task, can another model find the decisive mistake and fix it with a single action? We introduce ReplayRCA, a benchmark that scores such diagnoses by intervention rather than by agreement with annotations: the environment is restored to the state preceding the chosen step, the proposed action replaces the agent's own, and the same agent continues. The score, causal repair, is the continuation success rate minus that of a paired placebo that replays the agent's original action through the same mechanism. Two tracks share this protocol: a post-hoc autopsy, where the diagnoser sees the whole failed run and is told that it failed, and live oversight, where a supervisor watches the run unfold and may intervene once, before an irreversible action, without knowing whether the run will fail. Tasks come from OpsWorld, a deterministic, procedurally generated back-office environment with six difficulty levels, the last of which is multimodal (QR codes, handwriting). On 62 calibrated systematic failures, the best of 13 frontier models (Grok 4.7) reaches a causal repair of +51.3 points; the top eight models are statistically indistinguishable, the flagship models of the two largest vendors rank 8th and 12th, and 5 items are repaired by no model. Forcing a strong model to intervene at the last step costs it 17.1 points, so the last-step shortcut of annotation-based attribution benchmarks does not transfer. On 87 runs, the two best live supervisors are statistically tied and one of them is a small open-weight model (DeepSeek V4.1 Flash); the supervisor that is among the best post-hoc diagnosers (GPT-6.1 Sol) is the weakest live one (-14.5 points on the same runs, p = 0.002), so post-hoc diagnostic performance can be a poor proxy for live oversight. The environment and the evaluation code are released at https://github.com/jeremy1392/RCAreplay.

Thu 29 OctArtificial Intelligence
The gist
When an AI agent messes up a complex task, can another AI figure out exactly where it went wrong and fix it with one action? The authors created ReplayRCA, a way to test this by letting a second model replace the faulty step and see if the agent succeeds afterward. Their tests show that spotting errors after failure and fixing them live are different skills, and the best models still can’t fix all mistakes. They also provide the environment and tools to help improve AI oversight and debugging.
Open → 2610.41025v1

Framework reveals patterns in large language model agent actions

Behaviour analysis for large language model-driven agents

Abstract: Understanding language-agent behaviour requires learning how actions are organised across a trajectory and how this organisation relates to context and outcomes. We propose a general framework that connects data-driven behavioural discovery with analysis and experimental investigation. Categories developed from recorded activity define what is measured. Analyses preserve the behaviours' positions in the process and express their relationships as interpretable hypotheses. Statistical assessment and tests in held-out contexts examine the strength and scope of the evidence. Experiments, including prompt steering, then test how behaviour and outcomes change under deliberately varied conditions. A study of iterative self-reflection illustrates how discovery, dependency learning and prompt comparisons can inform one another. We develop this approach for trajectories of varying length, event order and agent roles, and identify advances in coding, analysis and experimentation needed to support its wider use.

Fri 23 OctArtificial IntelligenceComputation and Language
The gist
Understanding how AI language agents behave is tricky because their actions happen over time and depend on context. The authors propose a framework that helps discover and analyze these behavior patterns from recorded data, making it easier to study how actions relate to outcomes. They also test their findings by changing conditions deliberately, such as tweaking prompts, to see how behavior changes. This approach works across different scenarios involving varying action lengths and roles. This helps improve how we analyze and experiment with AI agent behavior.
Open → 2610.32087v1

Pos signatures improve demonstration choice for dependency parsing

POS-Signature Demonstration Retrieval for In-Context Dependency Parsing

Abstract: Demonstrations for in-context learning are often retrieved by semantic similarity, which may be a poor proxy for relevance in structured prediction. We study demonstration selection for Universal Dependencies parsing with a POS-signature policy that represents sentences with weighted part-of-speech statistics and retrieves three length-compatible, gold-parsed examples from the labeled training split. We evaluate two open-weight language models without fine-tuning on all 2,077 sentences of the UD English Web Treebank test set, comparing the policy with shot- and format-matched fixed and semantic three-shot baselines, as well as zero-shot, chain-of-thought, and critique-refinement systems. POS-signature retrieval yields the highest labeled attachment score among the tested prompting systems for both models (72.4 and 73.2), outperforming semantic retrieval by 1.9 and 2.7 points; paired-bootstrap 95% confidence intervals exclude zero for both contrasts. Chain-of-thought and refinement comparisons vary across models. These exploratory results favor the tested POS-signature retrieval approach on English-EWT. We release the prompts, archived outputs, and evaluation code.

Thu 22 OctComputation and Language
The gist
When computers try to understand the relationships between words in sentences, they learn better if given good examples first. The authors found that picking examples based on parts of speech, rather than just meaning, helps language models do a more accurate job. They tested this on English sentences and showed it worked better than some other ways of choosing examples. They also shared their tools and results for others to use.
Open → 2610.31966v1

Gui agent improves task success by reusing subgoal memories

Subgoal-aware Memory Graph for GUI Agent

Abstract: GUI agents aim to automate complex user tasks by interacting with mobile interfaces, but ef- fective decision-making in long-horizon set- tings requires reusing prior experience rather than relying solely on the environment ob- servations. Existing memory-based methods typically store experience either at the single- action level or as entire trajectories. Single- step actions are often too fine-grained to carry meaningful semantics, since the success of an individual action does not necessarily imply progress towards the overall goal. In contrast, whole trajectories are often too coarse-grained and difficult to reuse across different tasks. We propose a subgoal-aware memory graph that distils past trajectories into structured, reusable evidence at the level of subgoals. We define a subgoal as consecutive actions shar- ing a single intent, the granularity at which progress toward the task becomes meaning- ful. To reduce redundancy and support cross- task reuse, we organise subgoal experiences as nodes in a graph, where related memories can be merged, extended, and retrieved based on the environment observation. On SPA-Bench, our method improves success rate by 11.11% over the baseline.

Thu 22 OctArtificial Intelligence
The gist
Automating tasks on mobile devices can be tricky because agents need to remember what worked before to make good decisions over many steps. The authors found that remembering every single action is too detailed, while recalling entire task sequences is too general. They created a system that remembers chunks of actions tied to specific smaller goals, called subgoals, and stores these in a graph that connects related experiences. This helps the agent reuse useful past knowledge to complete tasks better, improving its success rate in tests by over 10%.
Open → 2610.31307v1

Generative reward models improve rationale quality and reduce bias

Judging the Justification: Decision Certificate Supervision for Generative Reward Models

Abstract: Generative reward models (GenRMs) generate natural language rationales to justify preference judgments, yet a correct verdict does not guarantee a sound justification. Existing approaches often rely on reference explanations to supervise rationales, but obtaining such references can be costly and does not necessarily establish whether generated rationales support the resulting verdict. To address these limitations, we propose DeCert, a framework that evaluates the validity and verdict consistency of generated rationales without requiring reference explanations. Crucially, DeCert translates these assessments into preference supervision that accounts for both verdict correctness and justification quality. In particular, when multiple sampled judgments share the same correct verdict, differences in certificate quality enable DeCert to construct rationale preference pairs that distinguish stronger from weaker justifications. Extensive empirical studies across two model families and multiple in- and out-of-distribution benchmarks demonstrate that DeCert consistently outperforms strong generative judge baselines while improving rationale quality. Moreover, DeCert effectively mitigates positional bias, improving position-consistent accuracy by up to 14.8 points. These results highlight the importance of supervising not only which verdict is correct, but also how that verdict is justified.

Tue 20 OctComputation and Language
The gist
Decision-making AI can explain why it makes certain choices, but sometimes these explanations don’t actually support the decision. The authors propose a new approach called DeCert that judges how good these explanations are without needing perfect example explanations to compare against. DeCert helps AI systems prefer explanations that better justify their choices, leading to clearer and more trustworthy reasoning. Tests show DeCert makes these explanations more consistent and reduces errors caused by bias.
Open → 2610.27528v1

System one models continuously monitor and guide system two reasoning

SOMA: Continuous Monitoring and Intervention for System Two Reasoning with System One Models

Abstract: Reasoning models often run into problems without noticing them, even when symptoms are visible in their own text. In humans, metacognition separates monitoring from control: fast, intuitive signals keep watch while slow, deliberate reasoning unfolds and help decide whether to continue, change course, or stop. System One (S1) decision models, such as Jev, are a natural candidate for this role: they return a calibrated probability for a natural-language question about a text in one fast call, yet they are used only as gates before reasoning. We introduce SOMA (System One Metacognitive Awareness), in which an S1 model continuously monitors a System Two (S2) reasoning model and intervenes when needed. SOMA has three layers: a scenario layer states each objective in natural language with predefined actions, an awareness layer lets S1 judge whether the described behavior appears, and a control layer accumulates these judgments, first reminds S2, and takes over only if the behavior persists. Sharing one S1 model and one controller across tasks, without training, SOMA addresses two different failures on two reasoning models: on overthinking, it cuts reasoning tokens by up to 36\% while improving average accuracy, and on unanswerable questions, it raises abstention from 70.5\% to 87.2\%.

Mon 19 OctArtificial Intelligence
The gist
Sometimes computer reasoning systems think too much or give answers when they shouldn't. The authors created a way for a fast, simple model (system one) to watch a slower, careful model (system two) as it works. This watcher can remind the slower model to stop or adjust if it sees mistakes or thinking going on too long. This method cuts down unnecessary thinking and makes answers more accurate or lets the system wisely say it doesn't know.
Open → 2610.26793v1

Tool using agents improved to handle weaker user instructions better

Not Every User Says It All: Training Tool-Using Agents Beyond Ideal Simulated Users

Abstract: Tool-using agents are trained and evaluated almost exclusively with strong simulated users who state every detail clearly, yet real users are often weaker. We construct weak users by masking sentences of a simulated user's replies, holding the task and the user's knowledge fixed, and cross-evaluate agents trained with strong and weak users on tau2-bench and ToolWOZ. Weak users lower the success of every policy, and training with them does not help: no policy trained with weak users, through fixed masks, mixtures or a curriculum, serves weak users better than the strong-trained policy beyond run-to-run noise, and prompting the agent to ask more does not help either. We propose StaGPO (State-Grouped Policy Optimization): the agent writes what the task needs, what is missing and how it is being filled, and GRPO uses this state to compare turns of different rollouts that reach the same situation, assigning step-level credit. Within the standard recipe of a small SFT cold start followed by RL, StaGPO makes weak-user training useful: on ToolWOZ, the Qwen3.5-2B policy trained with weak users reaches 76.8% with weak users, the highest of any Qwen3.5-2B policy we trained, 4.8 points above GRPO and 2.8 above the strongest baseline; it is also 5.4 points above GRPO with strong users, and generalises to an unseen mask rate; trained with strong users instead, it serves weak users at least as well as GRPO trained with weak users. On tau2-bench it scores highest with weak users for both models, by a small margin for Qwen3.5-4B.

Fri 16 OctArtificial IntelligenceComputation and Language
The gist
Many AI agents that use tools are trained assuming users always explain their needs clearly and fully. The authors show that when users provide weaker or incomplete instructions, these agents perform worse, and simply training on weaker users doesn’t help much. They introduce StaGPO, a new training method that helps agents track what is known and missing during a task, improving their success with weaker users. Experiments on two benchmarks show that agents trained with StaGPO work better with weaker users, even generalizing to new situations.
Open → 2610.24294v1

Splitting test-time compute optimizes model answer verification accuracy

Sample, Mix, or Verify? Allocating Test-Time Compute under Imperfect Verifiers

Abstract: A common way to make language models and agents more accurate is to spend more compute at inference time: generate many candidate solutions and submit the one a verifier prefers. Success then requires that some candidate is correct and that the verifier picks it. The first grows predictably with the number of samples; the second does not, because real verifiers (unit tests, majority votes, LLM judges, reward models) accept wrong answers, reject right ones, and make correlated mistakes. With a fixed budget, a practitioner must decide how many samples to draw, which models to draw them from, and how much to spend on verification, and today these choices are made by rules of thumb. We introduce SPLIT, which makes them from a small labelled pilot. It rests on an exact formula for the probability of submitting a correct answer under accept/reject and top-score selection that accounts for varying task difficulty, blind spots shared across models, and correlated repeated verifier calls. SPLIT fits the formula by empirical Bayes and uses it to plan budgets on unseen tasks. Across seven benchmarks spanning math, science QA, code and software-engineering agents, 26 models, 175 agent systems and six kinds of real verifiers, all from public releases, SPLIT lands within 0.02-0.8 percentage points of the best allocation in hindsight, while every fixed rule we test loses at least 2.5 points in some setting. Regret depends only on how well SPLIT ranks configurations, and it stays near-optimal under compute, price and latency costs and online. The formula also explains when asking more LLM judges lowers precision. As a forecaster far beyond the pilot, SPLIT beats power-law fitting when the verifier caps success, and we report where it loses.

Fri 16 OctMachine LearningComputation and Language
The gist
When AI systems check their own answers, they use some process to pick the best one, but these checkers make mistakes and sometimes miss the right answer. The authors introduce a method called SPLIT that figures out the best way to spend limited computer power on generating answers and checking them, based on a small test run. This approach predicts which setup will work best even on new tasks, improving overall accuracy closer to the best possible with known hindsight. It also explains why asking more human-like judges sometimes reduces accuracy instead of helping.
Open → 2610.23255v1

Graph method reveals where AI agents act less consistently

GraphACM: Graph-based Action Consistency Measurement for Uncertainty Quantification in LLM Agents

Abstract: Behavioral consistency across repeated executions provides an informative signal for assessing the reliability of large language model (LLM) agents. However, existing approaches typically summarize consistency with a single task-level score, without identifying where behavioral differences occur or whether actions are compared under equivalent decision contexts. We propose a graph-based framework for fine-grained behavioral consistency measurement and uncertainty quantification. Given multiple rollouts of the same task, we merge equivalent states into a directed, action-labeled graph, enabling action comparisons at shared decision states. We introduce node-level and global measures of action consistency, using the global consistency score as a proxy for uncertainty to predict task success or failure. Experiments across three LLMs on ALFWorld, AppWorld, and WebShop demonstrate that our measure is positively associated with task success and achieves an average AUROC of 0.811 for task-success prediction, outperforming the strongest behavioral-consistency baseline (0.729) and verbalized confidence (0.769). These results highlight the effectiveness of state-conditioned behavioral consistency and pinpoint where agent actions agree.

Thu 15 OctArtificial Intelligence
The gist
Large language model agents sometimes make different choices when asked to do the same task multiple times, making it hard to trust their results. The authors created a way to compare their decisions in a detailed, step-by-step manner using a graph that represents different states and actions. This method helps identify exactly where the agent’s behavior changes and measures how confident we can be about its correctness. Tests show that their approach better predicts when the AI will succeed or fail than previous methods. This means developers can better understand and improve AI agent reliability.
Open → 2610.22031v1

Large language model agents share experience to improve task success

XferAgent: Cross-Agent Experience Transfer for Self-Improving LLM Agents

Abstract: Large language model (LLM) agents can reuse experience from prior task executions to improve performance on subsequent tasks. Although existing methods enable cross-agent experience sharing, differences in agents' execution histories and task requirements make shared experiences difficult to reuse effectively. We propose XferAgent, a framework for cross-agent experience transfer that organizes reusable experience in local and shared experience graphs. XferAgent reconciles compatible contributions from heterogeneous agents while avoiding unreliable merges. For each new task, it selects shared experiences and integrates them with the target agent's local experience to support task execution. Experiments across diverse task domains show improvements over representative memory-based methods and effective experience transfer between heterogeneous agent frameworks. Cumulative task success also increases over successive task passes as agents update and reuse their experiences.

Wed 14 OctMultiagent SystemsArtificial Intelligence
The gist
Large language model agents sometimes repeat tasks without getting better at them. The authors introduce XferAgent, a system that helps these agents share their past task experiences with each other in a careful way. This sharing helps agents learn from one another even if they have different backgrounds or ways of working. Tests show that using this system leads to better success on tasks over time as agents reuse and build on shared knowledge.
Open → 2610.19581v1

LLM agents struggle with user emotions in long tasks

$τ$-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions

Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Yet existing benchmarks largely assume cooperative and behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. We introduce $τ$-Emotion-Bench, a ten-domain benchmark for evaluating tool-calling agents under controlled emotion-conditioned user interactions while preserving the underlying task objective. To scale both benchmark construction and agent training, we introduce Tracer, a scalable agentic data-generation pipeline that constructs grounded, policy-valid tasks and collects trajectories through an interactive agent--user--environment harness. By allowing dissatisfied users to terminate inefficient interactions, the harness provides an interaction-aware signal that filters trajectories with excessive user dependence and favors effective tool use. Experiments with two user simulators show that frontier agents remain unreliable, with substantial gaps between average task success and consistent task completion. Tracer-generated supervision consistently improves Qwen3 models across scales, generalizes to unseen domains, and transfers to $τ$-Bench and $τ$-Trait. Behavioral analyses further show that higher task success is associated with a better balance between tool execution and necessary user follow-up. Overall, $τ$-Emotion-Bench and Tracer provide a scalable framework for evaluating agent reliability under diverse user behaviors and improving tool-calling agents through interaction-aware supervision.

Wed 14 OctArtificial IntelligenceComputation and Language
The gist
LLM agents that use tools to help users can fail not just because they misuse the tools, but also because they don't handle different user emotions well. The authors created a new benchmark called τ-Emotion-Bench to test these agents with users showing various emotions, making it more like real life. They also developed a method called Tracer to generate many example interactions and train better agents. Their tests show that agents still have trouble consistently completing tasks when users get frustrated or emotional. The study suggests that better balance between using tools and managing user follow-up improves success.
Open → 2610.19560v1

Large language models adjust creativity based on conversations context

Read the Room Before Your Model Generates

Abstract: Large language models (LLMs) generally rely on sampling temperature to regulate the amount of randomness in their outputs. Mainstream methods for selecting temperature typically involve fixed, heuristic global settings and context or token-level calibration, rather than a geometrically interpretable representation of the communicative scenario. In this work, we introduce a simple and interpretable temperature selection framework, the \textbf{Scenario-Actor}, which adapts decoding to the inferred situational characteristics. Scenario-Actor extracts scenario-relevant structures from Transformer hidden states and determines the temperature from scenario embedding, a structured geometric representation of the communicative setting. Empirical results show that on Llama-3.1-8B-Instruct, Scenario-Actor achieves strong goal-completion performance in interactive social scenarios, significant gains on open-ended instruction following, and competitive performance on multi-turn conversation compared to multiple baselines.

Tue 13 OctArtificial IntelligenceComputation and Language
The gist
Large language models often decide how random or creative their replies should be by using something called temperature. Usually, this temperature is set by simple rules that don’t really consider the whole conversation or situation. The authors created a new way to figure out this temperature by understanding what's happening in a talk, called the Scenario-Actor method. This method helps the model pick a better temperature based on the kind of conversation, making responses more suitable and effective. They tested this approach on a model called Llama-3.1 and found it did better in chatting and following instructions.
Open → 2610.18820v1

Agent reflex improves language agents reflection and recovery in noisy settings

Agent-Reflex: Teaching Language Agents to Act Reflectively

Abstract: Large language models have evolved from single-turn chatbots into interactive agents capable of long-horizon reasoning. In real deployments, MCP tool services can return noisy feedback and users may intervene to correct the agent, but existing methods provide limited supervision for reflection learning and failure recovery. To bridge this gap, we propose Agent-Reflex, a general training framework for incentivizing proactive reflection in agents. Specifically, Agent-Reflex comprises three stages. First, Environment Reflection Skill Discovery (RSD) injects grounded tool-feedback perturbations into environment scaling, samples a capability-diverse LLM pool, mines matched success-failure pairs, and summarizes reusable reflection meta-skills. Building on this skill library, we then apply Reflective Supervised Fine-Tuning (ReSFT) that converts each skill into supervision for self-reflection, skill recovery, and user-centric correction, strengthening complementary reflective behaviors. Finally, we propose On-Policy Reflective Policy Optimization (ORPO) to combine agentic reinforcement learning under noisy interactions with reflection-guided on-policy distillation to further elicit proactive reflection. Built on Qwen3-8B, Agent-Reflex reaches 58.6 overall on five long-horizon tool-use benchmarks and 29.1 on AgentNoiseBench, surpassing strong environment-scaling and reflection baselines. Experiments on six challenging agent benchmark suites demonstrate the effectiveness of Agent-Reflex. Further analyses show that Agent-Reflex improves agents' robustness to noise while maintaining stable training dynamics, supporting scalable agent training. Our code is available at https://github.com/RUC-NLPIR/Agent-Reflex.

Tue 13 OctComputation and Language
The gist
Some AI language agents struggle when their tools give confusing feedback or when people try to fix their mistakes. The authors developed Agent-Reflex, a way to teach these agents to think carefully about their errors and fix themselves more proactively. It works in three steps: discovering reflection skills from past successes and failures, training the agent with these skills, and then refining the agent using reinforcement learning while it interacts. This method made agents better at handling tricky tasks, recovering from errors, and dealing with noisy responses.
Open → 2610.17953v1

PaceReasoner boosts short answers accuracy using long reasoning

PaceReasoner: Label-Free Learning to Answer Briefly from Long Reasoning

Abstract: Extended chain-of-thought reasoning improves problem-solving performance but increases inference cost and latency, motivating more accurate answers with shorter reasoning. We introduce PaceReasoner, a label-free test-time reinforcement learning framework that uses a model's LONG responses to improve its SHORT-mode accuracy. Both modes share a trainable policy, creating a need for reliable supervision, sufficient SHORT training samples, and continued LONG responses to guide learning. PaceReasoner combines all-response and LONG-only answer consensus through progressive, confidence-weighted supervision. A behavior reward gradually shifts from encouraging SHORT exploration to rewarding agreement with LONG answers, while a ratio controller regulates the balance between modes. Across three model backbones and four benchmarks, PaceReasoner improves average SHORT accuracy over the baselines by 7.4--13.5 percentage points, with fewer than 25 reasoning tokens per response on average. Ablations support the key components, and cross-dataset results demonstrate transfer beyond the adaptation set.

Tue 13 OctArtificial IntelligenceMachine Learning
The gist
Answering complex questions often requires long step-by-step thinking, but that takes more time and computing power. The authors introduce PaceReasoner, a method that helps a model learn to give better short answers by using its own longer explanations as guidance. It does this without needing extra labeled examples, balancing between short and long thinking modes during learning. Their tests show PaceReasoner improves short answer accuracy by a significant margin while keeping answers brief.
Open → 2610.17900v1

Visual language models learn better by using their own mistakes

The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors

Abstract: Visual in-context learning (ICL) enables vision-language models (VLMs) to adapt at test time via demonstration examples, but its effectiveness depends critically on the examples selected. Recent counterfactual selection methods improve ICL by constructing demonstrations that expose how individual attribute changes affect the answer. However, they require multi-stage pipelines involving attribute extraction, caption engineering, and composed image matching. We propose a simpler alternative grounded in a key observation: a model's own prediction errors are natural counterfactual demonstrations that directly encode its failure modes. Our method, SMILE (Self-Mined Visual In-Context Learning from Errors), builds a model-specific pool of hard negatives offline by recording the target VLM's own mistakes. At test time, the prediction serves as a direction signal that surfaces, in the semantic embedding space, the past mistakes most likely to recur for the current query. This prediction-conditioned identification provides a unified mechanism for both closed-set classification and open-ended visual question answering without task-specific design. Across four benchmarks and four VLMs, SMILE delivers an average task-score gain of +3.09 points over the SOTA baseline at 5x lower latency. Project page: https://tree-shu-zhao.github.io/SMILE/

Mon 12 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Choosing good example images helps vision-language AI make better predictions, but it's usually complicated to find these examples. The authors found that the AI’s own mistakes are actually useful because they show where it tends to go wrong. Their method collects these mistakes and then uses them to improve answers without extra complicated steps. This approach works well on different tests and models, making it both faster and more accurate than previous methods.
Open → 2610.16977v1

Pid gates improve prompt updates under noisy feedback

PID-Gated Continual Prompt Adaptation Under Noisy Feedback

Abstract: Deploying LLM agents in non-stationary environments requires continually revising their instructions as the task drifts, yet rewriting after every reported mistake can turn noisy feedback into persistent errors. This raises an often-overlooked scheduling question in prompt adaptation: when should feedback trigger a lasting instruction update, and at what editor-call cost? Taking inspiration from proportional-integral-derivative (PID) control, we introduce a family of lightweight gates that react to the current error, accumulate evidence over time, and roll back deterioration after an edit. We benchmark these gates against both simple baselines and advanced prompt optimizers (TextGrad and GEPA), over a diverse suite of drifting tasks, LLM parameter sizes, feedback-noise levels, and random seeds. PID-gated continual prompt adaptation yields a more favorable Pareto trade-off between task performance (high accuracy, low error) and editor-call budget than the baselines considered, pointing to control-theoretic scheduling as a simple, general tool for robust prompt adaptation under unreliable feedback.

Mon 12 OctMachine LearningArtificial IntelligenceComputation and Language
The gist
Large language models (LLMs) need to update their instructions as tasks change, but relying on every single correction can cause mistakes to stick, especially if those corrections are noisy or wrong. The authors propose using control theory ideas from PID controllers that help decide when to update instructions based on error, accumulated evidence, and ability to undo bad changes. Their approach, called PID-gated prompt adaptation, balances making accurate updates while limiting costly edits, performing better than simpler or more complex alternatives in varied noisy scenarios.
Open → 2610.16129v1

Benchmark shows agents struggle with real business task delivery

FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests

Abstract: Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience. Forward-deployed engineers (FDEs) must elicit these requirements through dialogue and iteratively optimize solutions until they meet customer acceptance criteria. Existing optimization benchmarks typically start from explicit objectives, while clarification benchmarks often introduce information gaps into existing tasks. We introduce FDE-Bench, the first benchmark built from real customer projects to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization. Its 49 cases span combinatorial optimization and machine learning. Clarification targets are distilled from discussions between frontline practitioners and FDEs. Agents elicit missing requirements from a simulated customer, and case-specific evaluators assess final artifacts against acceptance criteria using customer-accepted FDE solutions as quality references. Across 18 agent configurations, the highest mean delivery score is only 0.5510, compared with an FDE reference of 1.0. The highest delivery success rate (DSR), defined as the fraction of planned runs reaching this reference, is only 6.12%. These results show that reliable FDE-level end-to-end delivery remains out of reach for the evaluated agents on underspecified real-world business tasks. The benchmark project page is available at https://baidubce.github.io/FM-Agent/FDE-Bench/.

Sat 10 OctArtificial IntelligenceMultiagent Systems
The gist
Many business requests don’t clearly explain what is needed because workers rely on experience they can’t fully describe. The authors created a benchmark called FDE-Bench using real cases where engineers have to talk to customers to figure out missing details and improve solutions step-by-step. They tested different AI agents on these tasks, but none performed as well as human engineers. This means current AI still finds it hard to fully handle complex business tasks where requirements are unclear.
Open → 2610.14539v1

Latent world model improves gui agent speed and success rates

Latent World Action Model for GUI Agents

Abstract: Pixel-space world models have recently emerged as a promising approach for GUI agents, predicting future screens prior to execution and subsequently re-perceiving them for action selection. However, this two-stage paradigm faces two key challenges in GUI environments: (L1) the high computational cost of pixel reconstruction and (L2) the need to reconstruct large and diverse appearance-level details beyond what is required for action selection. In practice, this screen-generation objective can induce an identity shortcut problem, where future-state prediction largely preserves the current state rather than capturing action-induced changes. In this work, we reformulate GUI world modeling by shifting the prediction target from full-screen reconstruction to action-induced future information in latent space. Concretely, we leverage a large multimodal model as the latent world model, whose shared representations support two dedicated heads for both action selection and predictive transition modeling. The action head directly translates these latent representations into action selection, while the transition head captures action-conditioned future information. To mitigate the identity shortcut, we further introduce an action-aware contrastive objective that encourages the transition representations to capture discriminative action-induced changes. Extensive experiments across offline and online GUI settings show that our method achieves up to 31.9% and 40.6% relative gains in success rate over pixel-space world models, respectively, while substantially reducing inference latency by up to 98.9%. Our code is available at https://github.com/sjyu001/latent2world.

Fri 9 OctComputer Vision and Pattern RecognitionArtificial IntelligenceHuman-Computer Interaction
The gist
Predicting what will happen next on a computer screen is hard and slow when trying to recreate every pixel. The authors found a way to focus on the important changes caused by actions, instead of rebuilding the whole screen image. They use a system that understands the screen in a simpler, hidden form and predicts only relevant future changes. This method works faster and gets better at selecting the right actions to control graphical interfaces like apps.
Open → 2610.14228v1

Evaluating how AI agents admit uncertainty when facing conflicting knowledge

Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict

Abstract: When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusion? Existing evaluations of agentic systems focus primarily on task success, offering limited insight into how agents handle such conflicts. We propose to evaluate agents on epistemic humility (EH): the agent's willingness to recognize, act on, and communicate uncertainty during task execution. We operationalize EH through three trajectory-level behavioral dimensions: Identify, Solve, and Escalate (ISE). Through knowledge conflict, situations where the backbone language model's parametric knowledge contradicts the evidence it encounters, or where two contextual sources disagree, we evaluate two conflict settings: (1) controlled conflict and (2) naturally occurring conflict during multi-step agentic execution, each paired with matched no-conflict controls. Evaluating four agents, we find that higher task accuracy does not necessarily correspond to greater epistemic humility: some high-accuracy configurations recognize conflicts during execution but do not communicate unresolved uncertainty in their incorrect final answers. Trajectory-level analysis further reveals that agents frequently detect conflicts in early steps of execution but fail to maintain or resolve them in later steps. Finally, we show that model-level interventions can improve EH, but often at the cost of task accuracy, suggesting that epistemic humility emerges from the interaction among the backbone model, agent harness, and evaluation environment.

Thu 8 OctArtificial IntelligenceComputation and Language
The gist
This paper looks at how AI agents handle situations when what they 'know' conflicts with new evidence they find. The authors check if agents change their answers, admit they’re unsure, or stubbornly stick to wrong conclusions. They find that even when agents get answers right more often, they don’t always admit uncertainty or explain their doubts. Also, agents might notice conflicts early on but fail to resolve or communicate them properly later. Adjustments to the underlying models can help agents be more humble but sometimes make them less accurate.
Open → 2610.12360v1

Skill reranking improves with difference-focused text selection

SkillContrast: Difference-Guided Text Selection for Agent Skill Reranking

Abstract: Similar agent skills can share instructions but differ in their conditions of use. Query-based text selection may retain shared instructions and omit these distinctions. We introduce SkillContrast, a training-free selector that compares retrieved skills and retains their differing text with local context for a pretrained reranker. On 1,235 requests from SameCapRisk-Bench, it yields 54-72 more clean hits (requests that retrieve a helpful skill without its marked risky sibling) than TF-IDF query selection at identical per-candidate input lengths, across 2 retrievers and 2 reranker sizes. Length-matched component replacements identify differing text as the main contributor in the primary setting, with smaller, mixed context effects. Relative to full skill bodies, SkillContrast uses 51.1-58.8% fewer model-input tokens, with 10-18 fewer clean hits at 0.6B and matching or higher observed clean-hit counts at 4B. Candidate-relative differences thus complement query relevance in selecting compact reranking inputs.

Thu 8 OctInformation RetrievalArtificial Intelligence
The gist
Finding the right skill for a task can be tricky when many skills share similar instructions but differ in key details. The authors created SkillContrast, a way to pick out just the parts that are different between similar skills so a computer can better rank which skill fits a request. This method helps computers find more useful skills without mixing up risky ones, and it does this while using less text. By focusing on differences rather than all the shared information, SkillContrast improves the precision of skill selection.
Open → 2610.11650v1

Multi-agent ai systems improve negotiation reliability and deadlock recovery

Constitutional Gating and Deterministic Recovery for Multi-Agent LLM Negotiation: Ablations Against a Stateful Adversarial Gatekeeper

Abstract: Multi-agent LLM systems negotiating with a stateful counterpart waste model calls in three ways: polite loops that never meet the counterpart's hidden acceptance condition, malformed outputs that trigger retries, and compliance deadlocks in which the counterpart demands something the agent must refuse. We study a three-part control stack - a 5-Pillar runtime constitution, a 4-tier swarm (Director, three-agent majority vote, Monitor, schema hard gate) and Cognitive Annealing (deterministic deadlock detection, atomic purge of the agent-side context, a canonical recovery message) - against a released adversarial Gatekeeper whose acceptance rules are fixed regular expressions and whose LLM only renders reply text. The testbed has a known solution: it measures whether the stack executes a constitution-aligned strategy against swarm drift and recovers from deadlock, not whether it discovers anything. In five runs per configuration (30 runs; Gemini 2.5 Pro agents, Claude Haiku 4.5 Gatekeeper) we find: (i) the constitution and Director make an acceptable framing possible but not reliable - 0/5 baseline unlocks versus 1/5 and 2/5 with the constitution; when the swarm unlocks it does so in one turn with 7-8 calls and about 15k tokens (67-73% below baseline); when it does not, it costs 17-38% more; (ii) the Monitor and hard gate do not reduce unlocks and leave an audit trail; (iii) under a honeytrap-to-compliance deadlock, LLM-only steering escapes 0 of 5 times while atomic purge plus a canonical strike escapes 5 of 5 (Fisher $p = 0.008$) at the same call budget, with zero calls for the strike. LLM-written strikes failed the deterministic pre-flight 5 of 5 times although an LLM Monitor had approved four. Pre-registered hypotheses on average call and token reduction were not supported. Cost is bounded in every arm by deterministic stop rules; the stack adds recovery at no extra model cost.

Thu 8 OctComputation and LanguageArtificial Intelligence
The gist
Sometimes AI agents negotiating with a tricky partner get stuck in endless or broken conversations or demands they cannot meet. The authors studied a set of rules and controls to help these agents negotiate more smoothly and recover when stuck. They tested their system against a fixed-rule adversary and showed it can reduce wasted effort and escape deadlocks reliably. While some parts helped frame conversations better, only a specific reset mechanism reliably broke deadlocks without extra cost.
Open → 2610.11542v1

Latent space evaluation cuts AI judgment time by six times

Judging in Latent Space: Efficient Generative Reward Modeling via Semantics-Preserving Compression

Abstract: Reward modeling often requires jointly representing and reasoning over multiple evaluation criteria, yet verbalizing this process token by token can incur substantial inference cost. Recent work on latent reasoning suggests that continuous states may support this computation more compactly. We introduce LatentGRM, a latent evaluation framework built on semantic chunking, compression, and reconstruction. By using the structure of rubric-guided evaluations to guide compression, LatentGRM learns compact continuous trajectories that support autonomous pairwise judgments without generating textual assessments. A separate interpreter reconstructs evaluation text from these trajectories, providing an offline view of the information retained under compression. Under matched training data and backbones, LatentGRM achieves competitive aggregate preference accuracy relative to explicit Supervised Fine-Tuning (SFT) judges at both 4B and 8B scales. Across four benchmark domains, LatentGRM-8B compresses evaluation trajectories by 8.9--9.2x and reduces total judge inference time by 6.1--7.0x at vote@5. Controlled rubric interventions show that criterion-dependent preference information is carried through the latent sequence. Together, these results demonstrate that continuous latent evaluation can substantially reduce inference cost while preserving competitive judgment quality.

Wed 7 OctComputation and Language
The gist
Judging complex tasks often needs checking many things one by one, which can take a lot of computer time. The authors came up with LatentGRM, a method that turns these step-by-step checks into shorter, compact summaries in a special continuous form. This lets computers make decisions faster without writing out long explanations each time. Their tests show it keeps quality close to traditional methods but works much quicker. They also found that their summaries still keep important info for later review.
Open → 2610.09788v1

SearchJev speeds up search agents with better calibrated fast decisions

SearchJev: A Fast and Calibrated System-1 Model for Search Agents

Abstract: Search agents repeatedly make short decisions about relevance, evidence sufficiency, and search actions. Using generative language models for these decisions introduces latency and unreliable confidence. We present SearchJev, a fast and calibrated System-1 model that separates search decisions from System-2 reasoning and generation. Given a search state and a decision schema, SearchJev directly scores legal options without autoregressive output generation. We propose Soft-Label Learning for Calibrated Decisions (SLCD) to learn decision probabilities from uncertain supervision and calibrate their confidence. In a dual-system search agent, SearchJev handles short decisions and delegates uncertain judgments to System 2, which retains planning, query generation, and answer composition. We also introduce SearchDecision-Bench, a benchmark unifying six types of search decisions for training and evaluation. On SearchDecision-Bench, SEARCHJEV improves decision quality over same-size Qwen3.5 autoregressive models, achieves 5.2-5.3 times faster decisions, and reduces average expected calibration error by 41-74%. On BrowseComp-Plus, the dual-system agents achieve a 3.7-4.7 times speedup in active search time while improving answer accuracy from 45% to up to 54%.

Sun 4 OctInformation RetrievalComputation and Language
The gist
Search agents help computers decide quickly what information is relevant and what to do next. The authors built SearchJev, a fast decision-making system that separates quick choices from deeper thinking to work faster and more reliably. SearchJev uses a new way to learn and measure confidence in decisions, making it better at knowing when it should ask for help. Their tests show it makes decisions much faster and with better accuracy, improving overall search performance.
Open → 2610.05107v1

Vision based web agent improves accuracy on live websites interactions

WebFovea: When the Model Is Right but the Click Is Wrong -- Reliable Round Trips for Vision-Based Web Agents on Live Websites

Abstract: We present WebFovea, a vision-based web agent that placed 2nd in the WebRetriever Challenge 2026 with a final score of 57.0 out of 100. The challenge evaluates agents end to end on Protocol III of the WebRetriever benchmark (arXiv:2607.06118): starting from an entry URL on a live website, the agent must operate the site's own interface and return a verifiable answer. A capable multimodal large language model (LLM) is necessary for this, but not sufficient. The model's decisions reach the browser through the harness, the code between the model and the page. At every step, four things must go right: the model's reply must be parsed into the intended action, the action must take effect on the page, the result must be reported back accurately, and the model must be shown the information it needs. On real websites, many of the failures we observed occurred at one of these four stages rather than in the model's reasoning. A coordinate-space mismatch placed every click at 3/4 of its intended coordinates; actions on native dropdowns, inside iframes, and in text boxes failed silently; and self-generated chat-template tokens contaminated 4.9% of task episodes. WebFovea hardens each stage and surrounds the loop with guardrails that keep the agent within the rules and its budget. The four-stage view does not depend on the model, although some individual fixes do. Because we used the same model in all four submissions, the rise of our official hidden-set score from 31.0 to 57.0 reflects changes to the harness, up to run-to-run variance on live sites. We describe the design, the evidence for each component (including negative results), a failure analysis, the limitations, and a roadmap that includes routing different steps to different models.

Fri 2 OctMachine LearningArtificial IntelligenceComputer Vision and Pattern Recognition
The gist
Using AI models to interact with live websites is tricky because even if the AI knows what to do, the technical steps to carry out actions can fail. The authors created WebFovea, which focuses on making sure every stage of sending commands to a website, executing them, and reading results works reliably. Their system improved performance by fixing problems like misaligned clicks and hidden input errors. This approach helps AI better perform real web tasks by handling practical issues beyond just understanding what to do.
Open → 2610.03036v1

Agentic text to sql systems improve question handling to reduce hidden errors

Beyond Correctness: Resolving Underspecification in Agentic Text-to-SQL

Abstract: Agentic Text-to-SQL systems can interact with users to clarify underspecified queries before generating SQL. However, a correct execution result does not necessarily imply that the agent has adequately resolved the underlying underspecification: the agent may silently make unverified assumptions that happen to match the intended answer. We show that this behavior is driven in part by premature clarification termination. Although forcing an agent to ask more questions improves execution accuracy, ambiguities are concentrated in earlier interactions, making brute-force questioning inefficient. More importantly, even when explicitly prompted to plan its clarification process, the agent frequently abandons questions that it has already identified as relevant. To address this failure mode, we introduce PlanPool, which externalizes the clarification plan as a mutable question pool. Every planned question must be explicitly asked or dropped before submission, while newly discovered ambiguities can be added during interaction. Across three benchmarks derived from BIRD-Interact and Spider, PlanPool consistently improves ambiguity coverage and reduces silent failures over unconstrained and prompt-based alternatives, while maintaining competitive execution accuracy. Our results highlight an important distinction in agentic reasoning: identifying missing information is not sufficient, and the agent must also reliably maintain and resolve it before committing to an answer.

Fri 2 OctComputation and Language
The gist
Sometimes, systems that turn natural language questions into database queries don’t ask enough questions to clarify unclear requests. This can lead the system to guess answers that seem correct but may not truly match what the user wanted. The authors show that these systems often stop asking questions too soon or forget to ask important ones. They introduce a new method called PlanPool, which keeps track of questions the system plans to ask and ensures they are all addressed before giving a final answer. This approach improves how well the system covers unclear parts of a question and reduces silent errors while still answering accurately.
Open → 2610.02739v1

Agent control cuts memory use for assistant queries by 92 percent

APDMem: Agent-Controlled Progressive Disclosure for Query-Adaptive Long-Term Memory

Abstract: Personalized LLM assistants must recover sparse evidence from long conversation histories across queries of varying complexity. We introduce APDMem (Agent-controlled Progressive Disclosure Memory), a hierarchical long-term memory architecture that applies progressive disclosure to memory retrieval. Rather than relying on a flat memory store or fixed retrieval granularity, APDMem represents conversation history as four progressively detailed layers: thematic summaries, personalized key facts, turn-level evidence notes, and raw messages. At inference time, a controller applies progressive disclosure to the memory hierarchy: it first reads high-level summaries and drills into finer evidence only when needed. This creates an adaptive cost-fidelity trade-off: simple queries can terminate early, while complex temporal, multi-hop, or exact-evidence queries trigger deeper inspection. A note synthesizer converts retrieved evidence into a query-focused structure that consolidates facts, orders events, and flags contradictions before final answer generation. Experiments on LongMemEval show that APDMem achieves strong performance for long-context memory reasoning while accessing only 8% of the total conversations.

Thu 1 OctComputation and LanguageArtificial Intelligence
The gist
Answering questions using long conversations is hard because the right details may be buried deep in many messages. The authors created a system called APDMem where conversation memories are organized into layers, from simple summaries to full messages. When a question comes in, the system looks at summaries first and only digs deeper if it needs more details. This method helps find answers accurately without reading everything, saving time and computing resources.
Open → 2610.02472v1

Bayesian method improves confidence and trust in multiple AI models’ decisions

Counting Moves, Weighing Voices: Bayesian Dialectical Argumentation for Calibrated Multi-LLM Councils under Persistent Adversaries

Abstract: A multi-LLM \emph{council} lets several large language models (LLMs) deliberate on a question and return an answer together with a confidence estimate. As these systems become increasingly used for reasoning, that confidence should represent a calibrated \emph{probability of being correct}, and the decision should remain robust when some agents are persistently unreliable. Existing \emph{council aggregation} methods fail on both fronts: their confidence estimates measure decisiveness rather than correctness, and they cannot identify or discount persistently unreliable agents. We introduce Bayesian Dialectical Argumentation (BDA), which treats the council's \emph{typed} moves---who proposed, challenged, or conceded which answer---as observations of a classical annotator model with \emph{per-agent} reliabilities. This formulation recasts multi-agent deliberation as a reliability estimation problem, using the deliberation trace to infer agent reliability under persistent adversarial behavior. By weighting evidence according to inferred agent reliability, BDA yields calibrated posterior probabilities over candidate answers while allowing persistently unreliable agents to be inverted rather than merely outvoted. Across binary and multi-class benchmarks, BDA achieves the best calibration among zero-cost council aggregation methods, requiring no additional LLM calls, and improves robustness under persistent adversarial coalitions while remaining competitive in clean settings.

Thu 1 OctArtificial Intelligence
The gist
When several AI language models work together to answer questions, it’s important to know how likely their combined answer is correct. Existing ways to combine their answers tend to reflect how decisive the group is, not how accurate, and they struggle if some models are always unreliable. The authors created a new method that watches how these models argue—who agrees, disagrees, or changes their mind—and uses this to estimate how trustworthy each model is. This way, the combined answer comes with a better-calibrated confidence, and the system can handle models that may be deliberately misleading.
Open → 2610.02005v1

RAGScope controls risk and cuts cost in AI answer review processes

RAGScope: A Leakage-Controlled, Cost-Aware Evidence-Gating Protocol for RAG Hallucination Triage

Abstract: Retrieval-augmented generation (RAG) systems need inexpensive ways to route generated answers: accept low-risk outputs, review uncertain ones, and reserve strong verifiers for the expensive tail. We present RAGScope, a leakage-controlled protocol for evaluating local evidence gates that use only the task input, retrieved context, and answer text. The protocol combines context-grouped splits, fold-scoped preprocessing, group bootstrap intervals, deployment operating points, end-to-end runtime, and explicit source-shift stress tests. On three RAGTruth tasks, the enhanced gate RAGScope-E reaches 0.798 AUROC and 0.660 average precision (AP) in pooled grouped cross-validation. Its pooled AP exceeds ROUGE-L by 0.034 with a 95% context-group interval of [0.002, 0.064], although the AUROC gain is not significant and ROUGE-L remains stronger on data-to-text. At a top-10% review budget, RAGScope-E attains 0.748 precision; accepting the lowest-risk 50% yields 0.141 residual unfaithfulness. RAGScope-E runs in 6.22 ms/example on CPU, versus 145.75 and 223.07 ms/example for the tested DeBERTa-NLI and HHEM settings. A 14,900-example HaluBench stress test exposes the deployment boundary: an in-domain calibrated gate reaches 0.879 AUROC, but leave-source-out calibration averages only 0.466. Target-only calibration recovers to 0.675 AUROC with 100 labels per source and 0.685 with 200. Cheap evidence gates are therefore useful routing components, but learned calibration must be validated and adapted within the target domain.

Wed 30 SeptCryptography and SecurityArtificial Intelligence
The gist
Sometimes AI systems that answer questions using extra information can make mistakes or 'hallucinate' facts. The authors developed RAGScope, a way to decide when to trust an AI's answer and when to ask for a review, without needing expensive checks every time. This method uses only the question, the information retrieved, and the answer to quickly pick out risky responses for closer inspection. Tests showed it is faster and nearly as accurate as more complex methods but needs careful adjustment when used with new kinds of data.
Open → 2609.39075v1

Hierarchical reinforcement learning improves coding agent efficiency

Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents

Abstract: Recently, coding agents have emerged as a dominant paradigm for real-world software engineering (SWE) scenarios, which solve complex tasks through multi-turn interactions with development environments. However, frequent interactions with environments would inevitably introduce substantial token overhead, leading to high usage costs and latency. Although recent studies have explored reducing token usage by context manipulation and interaction limits at inference time, these approaches focus on improving token efficiency while overlooking the risk of discarding task-relevant information, thus struggling to balance the trade-off between resolution rate and token efficiency. In this paper, we study a more general paradigm without suffering from the limitation, i.e., training token-efficient coding agents with promising resolution performance, which is a highly-practical yet less-explored problem. To this end, we reveal two core observations in SWE scenarios: i) Efficiency Variation: successful resolution could be achieved with fewer tokens; ii) Entropy Correlation: unproductive behaviors are associated with turn-level entropy. Motivated by observations, we propose a novel reinforcement learning framework, dubbed HERO. Specifically, HERO prioritizes task resolution over token efficiency during policy optimization and encourages efficient reasoning patterns at both trajectory and turn levels. Extensive experiments on SWE-bench Verified and SWE-bench Multilingual demonstrate that HERO achieves a favorable trade-off between resolution rate and token efficiency compared with state-of-the-art coding agents and reinforcement learning methods.

Wed 30 SeptSoftware Engineering
The gist
Coding agents help write software by interacting with development tools, but they often use too many tokens, which costs time and money. The authors found that some tasks can be solved with fewer tokens and that wasted effort is linked to unpredictable actions. They created a new learning method called HERO that teaches agents to focus on solving tasks efficiently without losing important details. Tests showed HERO balances task success and token use better than previous methods.
Open → 2609.38885v1

Llm agents improve understanding despite user miscommunication and changing goals

Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent

Abstract: Modern LLM agents increasingly tackle complex tasks through interactive, long-horizon exchanges with users, while existing benchmarks generally assume that users always accurately and sufficiently communicate a fixed intent. However, this oracle communication assumption rarely holds in practice: users may miscommunicate, change their goals, and run out of patience. We define this task setting as Interactive Intent Alignment, where agents must recover and continuously track the user's current intent despite imperfect communication and evolving goals. To study this setting, we introduce Drift-Bench++, a principled benchmark construction pipeline for verified executable tasks with controlled misalignment and intent shifts, along with an interaction protocol featuring finite patience, diverse simulated users, and silent interaction-conditioned shifts. We further develop GRIP, a comprehensive evaluation protocol covering task grounding, user realism, inquiry effectiveness, and adaptation to evolving intent. Across diverse environments, models, and interaction conditions, stronger interaction consistently helps but remains far from oracle performance; Validation on deployed ProdAgent sessions further shows that the modeled failures are prevalent and consequential in deployment. By providing a unified, executable benchmark for interactive intent alignment, Drift-Bench++ offers a foundation for evaluating and advancing agents under realistic communication and evolving intent.

Tue 29 SeptComputation and LanguageArtificial IntelligenceMachine Learning
The gist
Sometimes when people talk to AI helpers, they don’t explain what they really want clearly, or they change their minds along the way. The authors study how AI agents can better understand and keep up with these changing and unclear instructions. They made a new set of tests called Drift-Bench++ to see how well agents handle these tricky conversations. Their results show that while better conversations help AI understand users, there is still a big gap compared to perfect understanding. This research helps make AI assistants better at handling real-world chat where people don’t always explain themselves perfectly.
Open → 2609.38604v1

Advisors learn better advice for language models using self-distillation

AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

Abstract: A small trainable advisor can steer a frozen language-model executor using natural-language advice. In addition to learning from task rewards, the advisor can use feedback from completed interactions to improve its advice. However, a plausible correction need not change execution, yet learning from such corrections can still affect the advisor's future decisions in other contexts. In a shared-parameter model, we prove that such corrections can limit learning if their targets favor useful advice less strongly than those of other corrections. Keeping them less often than the rest improves the model's eventual performance compared to learning from every correction. Motivated by this, our method, Advisor Self-Distillation (AdviSD), pairs outcome-based reinforcement learning with self-distillation from a feedback-conditioned copy of the advisor selectively. Reflection proposes corrections, and the advisor scores the same recorded executor response with and without its issued advice, using the magnitude of the difference to select decisions for supervision. This approach does not require executor likelihoods or additional executor rollouts. Experiments with Qwen3-8B advisors for Gemini and Claude show that AdviSD outperforms advisor-GRPO by 4.2-6.4 percentage points on BFCL-v3 and by 3.9-5.1 score points on EnvScaler. The trained advisors generalize to out-of-domain tasks and transfer across different executor versions and model families. AdviSD also beats matched-count random selection, supporting the value of its selection rule.

Tue 29 SeptArtificial IntelligenceComputation and LanguageMachine Learning
The gist
Large language models can be helped by smaller advisor models that suggest what the big model should do next. This paper shows a way for the advisor to learn better advice over time by selectively learning from feedback about past advice, ignoring less useful corrections. The authors created a method called Advisor Self-Distillation (AdviSD) that improves the advisor by focusing on advice that really changes the big model’s behavior. Tests with popular language model advisors showed AdviSD gives more accurate and adaptable advice than previous methods.
Open → 2609.38142v1

SkillCome improves large language model skills with group contrast learning

SkillCome: Group Contrast Skill Optimization with Dual Memory

Abstract: Skill evolution improves the capabilities of large language models by analyzing trajectories generated under a given skill and modifying the skill accordingly. Existing approaches typically generate a single trajectory per question. However, this provides insufficient optimization signals since it requires inferring effective skill edits from a solitary path. It is difficult to pinpoint which actions caused the failure in a failed trajectory, or to determine which actions in a successful one should be incorporated into the skill. Furthermore, they rely on a local batch of trajectories for analysis, making the optimization direction susceptible to noisy evidence. To address these, we propose SkillCome, a Skill-evolution method based on group Contrast optimization with dual memory. For each question, SkillCome generates trajectories and performs group contrast analysis to precisely identify key behavioral divergences between successful and failed trajectories, offering reliable optimization signals. The dual memory system further accumulates evidence from historical steps to track patterns shared across different groups, leading to more generalized optimization directions. Together, SkillCome builds a systematic optimization process that transforms experience from observed successful trajectories into reusable skills. Extensive experiments on six benchmarks spanning question answering, reasoning, and agentic tasks demonstrate the effectiveness of our method. SkillCome consistently outperforms baselines across five models of varying families and scales, with gains up to +5.69 points.

Tue 29 SeptArtificial Intelligence
The gist
Large language models learn better by practicing skills and improving from their experience. Past methods only used one example of how a model solved a problem, making it hard to understand what worked or failed. The researchers developed SkillCome, which looks at groups of successful and failed attempts together to find important differences and improve learning. SkillCome also remembers past experiences to guide better learning overall. Tests showed it helps models do better at answering questions and reasoning.
Open → 2609.37128v1

Spatial credit assignment improves gui agent training from click feedback

SCA: Spatial Credit Assignment for Reinforcement Learning of GUI Agents

Abstract: GUI agents automate tasks on digital devices by grounding language instructions in visual interfaces. Existing group-relative reinforcement learning improves GUI action prediction by comparing the rewards of multiple responses sampled from the same GUI state. However, binary evaluation treats spatially different failed clicks as identical and provides no relative signal when all sampled clicks fail. To address these limitations, we propose Spatial Credit Assignment (SCA), which uses the screen coordinates of sampled clicks to refine group-relative credit. Specifically, SCA predicts each held-out response's reward from the other responses in groups containing both successes and failures, then uses the prediction residual to adjust credit. When all sampled clicks fail, SCA instead orders them by distance to the annotated target. These spatial references are used only to construct the training update; the deployed policy remains unchanged. We evaluate whether this correction improves the policy update itself by comparing its error and directional alignment with the exact return gradient in a controlled synthetic study. Across GUI grounding and offline action-prediction benchmarks, SCA improves grounding across professional domains and achieves the strongest results among reinforcement-fine-tuned models on most action-prediction metrics, with consistent gains across the reported GUI suites.

Tue 29 SeptArtificial Intelligence
The gist
Teaching computer programs to interact with graphical user interfaces (GUIs) is tricky when clicks miss the target. The authors found that treating all failed clicks the same way provides poor learning signals. They designed a method called Spatial Credit Assignment (SCA) that uses the position of clicks to better judge which attempts are closer to the goal. This helps the programs learn more effectively to interpret language instructions and perform GUI tasks accurately.
Open → 2609.36939v1

Failure-transparent agents reduce false success claims in AI tools

Failure-Transparent Agents: Benchmarking Post-Failure Reporting in Tool-Using Language Models

Abstract: Tool-using agents can fail twice: a required tool can fail, and the agent can then report success without the evidence needed to justify it. Existing benchmarks often entangle this reporting failure with tool selection, recovery, and environment dynamics. We introduce Failure-Transparent Agents (FTA), a controlled benchmark that fixes the failed observation and required evidence state before generation, making post-failure claims directly auditable. FTA contains 100 tasks with deterministic failure traces spanning five failure families, a neutral control, and four user-pressure conditions, and evaluates unsupported claims alongside useful recovery. Across six models, three response policies, and 3,600 human-annotated responses, false-success rates are 22.8% under the baseline policy, 9.3% with a transparency instruction, and 0.8% with a structured evidence contract. Fabricated-detail rates decrease from 28.3% to 14.3% and 0.8%, while useful responses increase from 74.9% to 89.2% and 98.8%, respectively. The tested evidence-contract policy is associated with substantially lower post-failure reporting errors while useful-response rates remain high within this blocked-task benchmark.

Mon 28 SeptArtificial Intelligence
The gist
Sometimes AI agents use tools that fail, and then the agents incorrectly say they succeeded without proper proof. The authors created a special test called Failure-Transparent Agents (FTA) to focus on checking whether agents honestly report failure after tools fail. They tried different models and methods, showing that adding clear instructions and requiring structured evidence drastically reduces false success claims and made responses more helpful. This work helps improve trustworthiness in AI that uses other tools.
Open → 2609.35732v1

Model uses confident answers to stop reading early and save time

The Model Knows When to Stop: Training-Free Early Stopping for Long-Context Reading

Abstract: Language models often process long inputs sequentially in chunks, but continuing to read after sufficient evidence has been acquired wastes computation. Existing stopping mechanisms either learn sufficiency from internal activations or train an exit gate, while a simpler alternative asks the model whether it has read enough. We introduce Answer-Convergence Stopping (ACS), a training-free stopping rule that measures rather than asks. After each chunk, it probes the frozen model's current answer state and stops when that state is both confident and stable. The rule requires only output-side generation and token log probabilities, has no trained components, and uses one shared configuration across models and benchmarks. Because a stopping policy can save computation simply by stopping too early, we evaluate the stopping decision itself using evidence position where available. On the full LongBench-v2 with two frontier models, ACS is the only stopping policy that matches or exceeds full-reading accuracy. Furthermore, across 250 S-NIAH questions, the premature stopping rate for ACS across five models from two families ranges from 0% to 12%, compared to 8.4% to 45.6% for the verbalized gate. Taken together, ACS reveals that by properly utilizing the output signals of frozen models, we can achieve favorable behaviors like adaptive stopping without the need for additional training.

Mon 28 SeptComputation and LanguageMachine Learning
The gist
Processing long texts can take lots of computer time if a language model always reads everything fully. The authors create a simple way for a model to decide when it has enough information by checking how confident and stable its answers are, without any extra training. This method, called Answer-Convergence Stopping, saves time by stopping early without losing accuracy. They tested it on tough reading tasks and found it works better than other methods that require extra learning.
Open → 2609.34590v1

Memory system for AI agents improves itself before errors happen

Remember Before You're Asked: MemDream for Self-Probing Memory Evolution

Abstract: Memory is essential for enabling LLM-based agents to maintain coherent, personalized behavior over long-horizon interactions. However, existing memory systems share a fundamental limitation: they never proactively test their own memory, repairing it only after real queries expose weaknesses. This reactive paradigm means every retrieval failure corresponds to a real interaction in which the cost has already been paid. We propose MemDream, a framework that enables self-probing memory evolution for LLM agents. Our framework periodically enters offline dream cycles where three specialized agents (Dreamer, Analyst, Consolidator) collaboratively probe, diagnose, and repair the memory graph before failures occur. A policy trained via Group Relative Policy Optimization learns which repair operations produce durable retrieval improvements, while a soft decay mechanism provides reversible forgetting driven by the same anticipatory signal. Experiments on LoCoMo and MemoryAgentBench demonstrate that MemDream improves answer F1 by 4.5 points on LoCoMo and achieves a 9.1-point higher overall score on MAB over the strongest reactive-evolution baselines.

Mon 28 SeptArtificial Intelligence
The gist
Language models that act as agents need memory to keep track of information over long tasks, but usually they only fix memory mistakes after they cause problems. The authors propose MemDream, a method where the system pretends to test its own memory when it's not actively working, finding and fixing issues early. This approach uses three roles working together to analyze and improve memory proactively, resulting in better answers and performance on tests compared to older methods. MemDream helps AI agents remember better before real mistakes occur.
Open → 2609.34545v1

Fixed language model judges make version-dependent mistakes evaluating agents

Frozen Judges, Moving Agents: Version-Dependent LLM-Judge Error and the Limits of Judge-Assisted Agent Evaluation

Abstract: Language-model judges compare agent upgrades with their predecessors, but a fixed judge can make version-dependent mistakes. We analyze 35 public coding-agent submissions (20 prespecified version pairs on 250 SWE-bench Verified issues), two customer-service agents (155 tau-bench tasks), and 1,106 expert-labeled AgentRewardBench trajectories. An upstream outage left three judges for the primary SWE-bench analysis (8,743 aligned agent-task cells); the fourth is descriptive. All three coding-agent judges and all four tau-bench judges reject task-conditioned error invariance after multiplicity adjustment. On SWE-bench, 32 of 60 judge-by-pair units have a detectable differential comparison component; eight judge-only intervals declare improvements that execution-based intervals cannot establish, despite rank correlations of 0.71-0.79. In tau-bench, one judge confidently reverses a nine-point reference-reward gap by penalizing a procedural habit the reward ignores. False acceptance of failed coding patches rises with agent capability conditional on task and reference outcome, while a task-solvability prediction from AgentRewardBench reverses sign in SWE-bench. Transporting old-version calibration raises mean absolute comparison error on SWE-bench from 3.8 to 19.5 percentage points; 24.6% of ratio-bootstrap draws are undefined near the correction boundary. A tuned paired audit narrows a classical interval by only about 5% at 80 labeled tasks. A randomized three-arm test does not support the predicted increase in false acceptance from showing the agent's final report (all three Holm-adjusted p-values = 1.0). These results favor explicit reference standards and paired audits of current outputs over judge-only release decisions or transported old-version calibration.

Mon 28 SeptMachine Learning
The gist
Evaluating improvements in AI agents using language-model judges that are kept the same over time can lead to errors that depend on which versions of the agents are compared. The authors studied multiple coding and customer-service AI agents and found that the judges often disagree depending on agent versions, sometimes wrongly approving inferior upgrades. They suggest that relying solely on these fixed judges or old calibration can cause significant mistakes. Instead, they recommend comparing agent outputs directly with explicit reference standards and current data audits.
Open → 2609.34198v1

Recursive skill evolution improves large language model agents performance

R$^2$ Flow: Recursive Self-Improvement via Recursive Skill Evolution

Abstract: LLM-based agents can improve themselves across tasks by reusing and revising the skills they orchestrate into executable procedures. Flow-based training fits this loop: it samples procedures in proportion to reward, and the flow through each skill credits it for the next library revision. Three obstacles stand in the way of making this self-improvement reliable: flow training suffers strategy collapse over tree-structured histories; nonnegative flow-based credit rewards frequent use as if it were benefit; and library edits rest on the task reward the policy optimizes. We introduce R$^2$ Flow, a recursive self-improvement framework that alternates policy learning, independent verification, and versioned skill-library updates on a shared-state orchestration graph. The graph merges histories that differ only in the order of independent steps, allowing flow training to pool evidence across equivalent executions. A flow-share readout of the trained flow, invariant to the backward policy, and a separate signed utility rank which skills to change, verifier evidence decides whether an edit is warranted, and a residual-variance plateau sets when to update. Committed edits reshape the graph the next policy learns on, realizing recursive skill evolution. Across question answering, mathematical reasoning, interactive decision making, and code generation, R$^2$ Flow improves task accuracy and library-edit precision over heuristic orchestration, reinforcement learning, and skill-evolution baselines, and transfers across executors. Code is available at https://github.com/beita6969/r2flow.

Sun 27 SeptArtificial Intelligence
The gist
Large language model (LLM) agents can get better at tasks by reusing and improving the skills they combine to solve problems. The authors present a method called R² Flow that helps these agents improve themselves step-by-step, by learning, verifying improvements, and updating their skill libraries more reliably. This method organizes how skills are combined so the system can share learning across similar task paths and decide more accurately which skills to change. In tests involving question answering, math reasoning, decision making, and code generation, R² Flow led to better accuracy and more precise skill updates than previous approaches.
Open → 2609.33867v1

Evolution-aware memory improves long-term AI agent interactions

EMIR$^2$: Evolution-Aware Memory with Intent-Guided Multi-Round Retrieval

Abstract: Long-term memory enables large language model (LLM) agents to leverage historical interactions for future tasks. However, existing memory systems struggle to utilize continuously evolving historical information, as they often rely on static memory representations and single-round retrieval strategies, failing to track factual changes or integrate distributed evidence across long-term interactions. To address these challenges, we propose \textsc{EMIR}$^{2}$, an \textbf{E}volution-Aware \textbf{M}emory framework with \textbf{I}ntent-Guided Multi-\textbf{R}ound \textbf{R}etrieval, enabling LLM agents to maintain evolving historical knowledge and adaptively retrieve relevant evidence. Specifically, \textsc{EMIR}$^{2}$ constructs a State-Evolving Memory Graph (SEMG) that represents long-term memory as evolving knowledge states supported by temporal event trajectories and evidential associations. By maintaining semantic states through evidence-based updates, SEMG preserves historical evolution and enables evidence tracing under complex and conflicting scenarios. Building upon this, we introduce an intent-guided multi-round retrieval mechanism that iteratively identifies missing evidence and expands retrieval based on accumulated information. Experiments on LoCoMo and MemConflict demonstrate that \textsc{EMIR}$^{2}$ improves long-term memory utilization, dynamic and static conflict handling, and complex retrieval performance, achieving relative improvements of more than 12\% in certain categories. These results highlight the effectiveness of jointly modeling memory evolution and adaptive evidence acquisition for long-term agent interactions.

Sat 26 SeptArtificial Intelligence
The gist
When AI agents try to remember things for a long time, their memory can get outdated or miss important details that change over time. The authors propose a new memory system called EMIR2 that keeps track of how information evolves and uses multiple rounds to find the right details. This system helps AI handle changes in facts and gather complex evidence better than before. Tests show it improves AI memory use and decision-making by over 12% in some cases.
Open → 2609.32584v1

Multi-agent systems manage changing tasks with issue tracking

RepoMAS: Solving Progressively Specified Tasks with Issue-Driven Multi-Agent Systems

Abstract: LLM-based multi-agent systems (MASs) have shown strong potential for solving complex tasks, but most assume that task requirements are sufficiently specified before execution. In practice, user requests are often incomplete, and additional requirements may only become clear during reasoning, tool use, or execution. We refer to such problems as progressively specified tasks. To systematically study this setting, we introduce ProgSpec, a benchmark that evaluates final outputs against requirements explicitly stated in the initial request and additional requirements supported by the available task evidence. We further propose RepoMAS, an issue-driven multi-agent framework inspired by open-source project management. RepoMAS records newly discovered requirements, conflicts, and failures as structured Issues and uses them to revise the task specification and execution structure during problem solving. Across ProgSpec and five existing benchmarks, RepoMAS achieves the best performance. Further analyses show that its issue-driven revision and repository maintenance mechanisms consistently contribute to performance. These results highlight the importance of allowing MASs to revise not only how a task is solved, but also revise their explicit representation of task requirements during execution.

Sat 26 SeptArtificial Intelligence
The gist
Tasks often change or get clearer while people are still working on them, but many computer systems expect all task details beforehand. The authors studied tasks that are only partly known at the start and become clearer during work. They created a new test that checks if a system can handle these changing needs well. Their system, RepoMAS, keeps track of new problems and changing requirements like a project manager would, updating its plan as it works. This approach performed better than others on several tests, showing that computer agents benefit from tracking and revising their understanding of tasks as they go.
Open → 2609.32490v1

Enterprise agents struggle with hidden facts in company data questions

Era by Eon: Benchmarking Enterprise Agents on Hidden Knowledge

Abstract: In the Era by Eon benchmark, each question states the rules for its answer, and code computes the answer from a generated company's data. When agents can run code, the four strongest models each answer 22 to 25 of 27 such questions, so the benchmark barely separates them. We add eight question templates that depend on hidden facts. No question or document states a hidden fact, and the records that seem to hold it show something else. Other data implies it. For example, the sales system says a customer dropped a purchase because of timing. On a recorded call, the customer blames an outage. For each generated company, code fills each template and computes an exact answer without a language model. We evaluate 12 agents. Each pairs a model with an agent program, which connects it to the company's systems. The best agent answers 18 of its 24 attempts, three per question, correctly. Four of the six models answer at most 6 of 24 with any program. The hardest questions require picking one of several similar records, such as which of three renewal offers a customer signed. All agents together answered two such questions correctly in only 1 of 84 attempts.

Thu 24 SeptSoftware EngineeringArtificial Intelligence
The gist
Some questions about company data have facts that aren't directly stated, making them hard for AI agents to answer. The authors created a test where agents must find answers relying on hidden clues, not just stated facts. They found most AI models handled easy questions well but struggled greatly with these hidden fact questions. This shows current AI tools have trouble understanding subtle or implied information in business settings.
Open → 2609.30055v1

Framework improves math tutoring by learning from past conversations

REAT: A Reflective Experience-Augmented Tutoring Framework for Multi-turn Mathematical Instruction

Abstract: Current Large Language Models (LLMs) excel at solving complex mathematical problems, yet this proficiency does not inherently translate into effective tutoring. While advanced LLM tutors may leverage multi-agent frameworks or fine-tuning, most still lack a mechanism to systematically accumulate and reuse pedagogical experience over time, limiting their adaptability to diverse student needs during fluid, multi-turn interactions. To bridge this gap, we propose the Reflective Experience-Augmented Tutoring (REAT) framework, which couples experience distillation from historical dialogues with real-time adaptive retrieval. Driven by a multi-agent Observer-Critic-Mentor (OCM) distillation pipeline, REAT reviews past conversational trajectories and distills raw interactions into structured, problem-agnostic pedagogical experiences. During live tutoring, a state-aware retrieval module injects these curated experiences to provide adaptive scaffolding based on the student's cognitive state. Experiments demonstrate that the proposed framework significantly outperforms both prompt-only and supervised fine-tuning (SFT) baselines, particularly in improving complex, low-scoring tutoring scenarios. Crucially, the distilled experiences exhibit robust generalization across diverse model architectures and mathematical datasets.

Thu 24 SeptMultiagent Systems
The gist
Solving math problems well doesn’t always mean a computer can teach math effectively. The authors designed a system called REAT that learns from previous tutoring chats and uses what it has learned to help students better during live sessions. It watches past conversations to find good teaching patterns and then uses those at the right moments, adapting to how each student is doing. Tests show that this method helps more in tough teaching moments than just giving the computer fixed instructions or retraining it.
Open → 2609.29804v1

Typed classifier judges answers cheaper and faster than llms with similar mistakes

JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

Abstract: We ask whether Jev, a typed classifier that returns probabilities over permitted answers without generating text, can replace an LLM rubric judge. We compare it with three flash-tier LLM judges on nine panels drawn from seven benchmarks, giving every judge identical criterion texts. Jev's accuracy differs significantly from an LLM judge's in only 8 of 27 paired comparisons, ahead mostly on binary criteria and behind only on graded ones, and most of the other comparisons are inconclusive. Summed over the nine panels, the LLM judges, called once per criterion, cost 29 to 325 times as much as Jev and took 30 to 220 times as long. On graded criteria all four judges agree more with one another than with the labels and mostly assign lower levels than the raters. One of several observational accounts is that raters followed scale conventions our criterion texts omit. Jev's confidence ranks its own errors on most panels, which should make a cheap classifier the ideal first stage of a cascade that defers its uncertain verdicts to an LLM judge. Correlated errors undo that advantage. The LLM judges repeat nearly all of Jev's most confident errors, so a cascade replayed on the recorded verdicts lowers cost but gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 even with oracle thresholds.

Thu 24 SeptComputation and Language
The gist
The authors investigate whether a typed classifier called Jev can be used instead of large language models (LLMs) to judge answers based on rubrics. Jev is much cheaper and faster than LLM judges and performs similarly in accuracy, especially on simple yes/no criteria. However, both Jev and LLM judges tend to make similar mistakes, especially on more graded, complex criteria. This means using Jev first and then asking an LLM only on uncertain answers saves cost and time but improves accuracy only a little.
Open → 2609.29769v1

SkillAA improves AI skill updating with precise graph-based editing

SkillAA: Attribution-Guided Skill-Graph Updating with Targeted Validation and Rollback

Abstract: External skills provide domain procedures without parameter updates, but existing methods often edit skills directly from failed rollouts without structured routing from an observed failure to an editable location; existing skill graphs also underuse semantic boundaries, object addresses, and topological dependencies for skill retrieval, targeted updating, and scoped validation. We introduce SkillAA (Skill Abductive Attribution), a structured skill-optimization framework for frozen language models. It represents skill applicability, execution, and composition in a unified graph, allowing the same structure to support skill selection, attribution-guided repair, and update validation. SkillAA contrasts successful and failed executions to route candidate repairs to specific graph objects, updates only the selected local structure, and uses Local and Big Gates to screen candidate changes before commitment. With gpt-5.6-sol, SkillAA reaches 81.5%, 66.7%, and 91.2% on SearchQA, LiveMath, and DocVQA, respectively, and attains the highest observed mean in every main setting. These results support the utility of attribution-guided graph editing and graph-scoped validation.

Thu 17 SeptArtificial Intelligence
The gist
AI systems often use fixed skills to perform tasks, but fixing errors in those skills can be tricky and inefficient. The researchers created SkillAA, a method that uses a detailed graph to represent skills and how they connect, allowing the system to locate the exact part that caused a failure and update only that part. This makes AI skill updates more accurate and reliable. Testing with a powerful language model showed SkillAA achieves high success rates on several question-answering tasks.
Open → 2609.20455v1

Multi-turn agents refined to cut cost and boost accuracy

Dependency-Aware Trajectory Refinement for Efficient Multi-Turn Agent Fine-Tuning

Abstract: Multi-turn agent trajectories often contain redundant rounds (failed tool calls, parallel sub-queries, verification-only steps) that inflate both training and inference cost. We propose viewing each trajectory as a \emph{round-level dependency DAG} that exposes which rounds are globally load-bearing for the final answer, and fine-tune agents on trajectories refined through this DAG. Given an LLM-annotated DAG, these edits are deterministic and interpretable, with optional rephrasing. Models trained on these refined trajectories consistently outperform those trained on the original trajectories at lower inference cost. Specifically, across four multi-modal QA benchmarks, our refinements improve downstream accuracy by up to $1.7$\,pp over vanilla SFT (and $5.7$\,pp over an LLM-deletion baseline) while reducing per-sample inference messages by up to approximately $40\%$ and inference tokens by up to approximately $48\%$, translating to substantial savings in compute and serving cost. Code is available.

Wed 16 SeptComputation and Language
The gist
When computers answer questions in multiple steps, some steps are unnecessary and waste time and resources. The authors suggest looking at these steps like a map that shows which ones really matter for the final answer. By teaching computers only with the important steps, they become better and faster at answering questions. This method was tested on several question-answering tasks and improved accuracy while reducing the time and computing power needed.
Open → 2609.18417v1

Speech language models improve reasoning accuracy with real time self correction

RetroThinker: Enabling Retrospective Thinking in Speech LLMs

Abstract: Speech large language models (SpeechLLMs) offer reduced latency and retain paralinguistic nuances that are typically lost in cascaded automatic speech recognition (ASR) and text-based LM architectures. However, they continue to lag behind text-only LLMs on complex reasoning tasks, while real-time spoken interaction imposes strict latency constraints. Although prior works employ Chain-of-Thought (CoT) and concurrent reasoning to enhance reasoning capabilities without inducing prohibitive delays, an inherent accuracy-latency trade-off persists. In this paper, we investigate whether a streaming SpeechLLM can dynamically revise its reasoning traces on the fly. We introduce RetroThinker, a multi-stage post-training framework that equips the Moshi model to self-verify and forward-correct CoT steps during inference. RetroThinker combines supervised fine-tuning (SFT) on curated retrospective thinking data with length-based direct preference optimization (DPO) to optimize retrospective during early reasoning (i.e., reasoning concurrently while the user speaks). Evaluated on the GSM8K benchmark, RetroThinker significantly improves the accuracy-latency trade-off over non-retrospective baselines, achieving an 11% absolute accuracy gain at a comparable latency.

Thu 10 SeptArtificial IntelligenceComputation and Language
The gist
Speech language models can understand spoken words faster and keep voice details better than converting speech to text first. But they are not as good as text-based models at solving tricky problems quickly. The authors created RetroThinker, which lets a speech model check and fix its own thinking steps while listening. This approach helps the model be more accurate without slowing it down much. Tests showed RetroThinker improved problem-solving scores by 11% with similar speed.
Open → 2609.11864v1

Weaker AI models improve task success with expert-guided corrections

Co-Evolving Harnesses and Models: On-Policy Correction Helps Weaker Models Catch Up Where Imitation Fails

Abstract: Agent harnesses (the system prompt, tool set, execution hooks, and context-management scaffolding around a model) are a critical determinant of agentic task success. Automated harness evolution can enable smaller models to perform well on domain-specific tasks at a fraction of frontier-model cost. Since both the harness and model weights shape behavior, we ask how harness evolution and lightweight fine-tuning should be combined. Across seven enterprise agent tasks, we first evolve a harness with the weaker model, then find that a stronger expert often uses it more effectively, suggesting expert supervision could close the remaining gap. However, training the weaker model on the expert's complete trajectories under the evolved harness backfires: performance regresses on all seven tasks by 4 to 30 points across Qwen3-Coder and Gemma 4, even though the same procedure helps under the unevolved harness. Our analysis shows that imitation transfers knowledge and increases scaffold usage, but disrupts model-harness fit: the weaker model adopts the expert's planning strategy without the competence to execute it and no longer matches the harness evolved around its native planning style. We therefore develop an on-policy expert-correction pipeline, automated by a meta-level MLE agent, that localizes the failing turn in the weaker model's own rollout and asks the expert to rewrite only that turn. This preserves the model's planning style and combines the gains of harness evolution and model adaptation. Our results identify and resolve a source of contention between harness and weight updates, yielding a compatibility-preserving recipe for economical co-evolution on domain-specific enterprise tasks.

Tue 8 SeptArtificial Intelligence
The gist
When smaller AI models try to do complex tasks, they rely on surrounding tools and instructions called harnesses to help them succeed. The authors found that just copying how a stronger AI model plans often makes the weaker model perform worse because the weaker model cannot do everything the stronger one does. To fix this, they created a method where the weaker model runs its own plan but gets help only on parts where it struggles, using corrections from the stronger model. This way, smaller models can learn better and work well with their own harness setup.
Open → 2609.09134v1

Memory cleaning improves task success for long running AI conversations

MeClear: Cooperative Game-Theoretic Attribution and Risk-Aware Memory Clearance for Long-Horizon LLM Agents

Abstract: Long horizon Large Language Model (LLM) agents rely on external memory systems to preserve user preferences and task knowledge across extended interactions. Conventional retrieval mechanisms optimize semantic compatibility rather than downstream utility, frequently introducing outdated, misleading, or conflicting evidence into the active context. We present MeClear, a task conditioned memory clearance framework that identifies memories featuring negative downstream utility through cooperative attribution and selectively suppresses them from agent execution. MeClear combines Leave One Out screening with sampled cooperative Shapley attribution to distribute utility across interacting evidence, effectively resolving redundant conflict masking where single removal evaluations fail. Utilizing attribution rankings, MeClear executes a query scoped minimal clearance strategy over a nested filtration, verifying task recovery on the cleared context without permanently altering the persistent memory bank. Comprehensive experimental evaluations across ten long dialogue memory pools demonstrate that MeClear achieves a target recall of 85.9% and an overall task recovery rate of 82.3%, representing a 25.5 percentage point improvement over Leave One Out (LOO) baselines.

Tue 8 SeptArtificial IntelligenceSoftware Engineering
The gist
When AI systems like chatbots remember too much over a long time, they sometimes use outdated or conflicting information that confuses their responses. The authors introduce MeClear, a new method that carefully checks which remembered information actually helps the current task and removes the parts that hurt the AI’s performance. By doing this in a smart way, MeClear improves how well AI agents complete complex tasks over long chats. Their tests show it works better than existing methods that remove bad memories one by one.
Open → 2609.09115v1

AgentGrad improves multi agent system prompts with guided fixes

AgentGrad: Intervention-guided Prompt Optimization for Multi Agent Systems

Abstract: Large language model (LLM)-based multi-agent systems (MAS) achieve strong performance by employing specialized multiple agents, yet their performance depends on the prompt design of each agent. For MAS prompt optimization, textual gradient methods that guide prompt updates using natural-language feedback have emerged as a leading paradigm. In this paper, we identify limitations in two stages of existing textual gradient approaches: gradient extraction and gradient aggregation. In gradient extraction, previous works select a target prompt without verifying whether modifying it resolves the failure, and derive gradients without agent-level supervision over the corresponding agent's intermediate output. In gradient aggregation, individual gradients are randomly grouped and concatenated, often mixing unrelated failure modes and producing prompts that fail to generalize. To address these limitations, we propose \textbf{AgentGrad}, a prompt optimization framework for multi-agent systems based on sequential intervention and semantic textual gradient abstraction. For each failure, sequential intervention modifies the behavior of one agent at a time to identify the target agent whose modification resolves the failure. The modified output of the target agent then serves as agent-level supervision for extracting a fine-grained gradient. Semantic textual gradient abstraction clusters semantically similar gradients to prevent mixing unrelated failure modes, and abstracts each cluster into a generalized gradient that captures the shared corrective pattern. Experimental results show that AgentGrad achieves state-of-the-art performance across five MAS benchmarks and reduces wall-clock optimization time by $2.5\times$ on average compared to the next-fastest baseline.

Tue 8 SeptArtificial Intelligence
The gist
Large language models work better when many specialized agents collaborate, but their success depends on how well each agent is told what to do. The authors found problems in existing ways to improve agent instructions because they guess which agent caused errors and mix fixes that don’t relate. They created AgentGrad, which tests agents one by one to find the one causing the problem and groups similar fixes for clearer improvements. This method works better and speeds up making agents smarter across several tests.
Open → 2609.08572v1