Papers for

customer support software developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Lightweight method improves diagnosis of AI agent failures with less cost

Lightweight, Rubric-Guided Trajectory Evaluation for Production AI Agents

Abstract: Trajectory evaluation is essential for improving the reliability of LLM-based agents, but production use makes it expensive to run repeatedly. Modern agents generate long traces containing tool calls, observations, retries, and external outputs, while not all raw tokens are equally useful for diagnosis. We present \textit{LiteTrajEval}, a lightweight architecture for budget-bounded trajectory evaluation. LiteTrajEval derives compact domain-specific rule profiles offline, then preprocesses each trajectory online, marks heuristic failure signals, serializes it under a fixed global budget, and invokes a single rubric-guided LLM judge to produce structured diagnostic reports. Evaluated on public Magentic-One-style and $τ$-bench-style trajectory datasets, LiteTrajEval improves failure-localization alignment with human annotations by roughly 20--35 percentage points on Magentic-One and up to 23 percentage points on $τ$-retail compared with AgentRx, while reducing cost by about 6$\times$ and evaluation time by more than 8$\times$. This solution has also been deployed in our enterprise agentic platform.

Fri 2 OctArtificial Intelligence
The gist
AI agents often make mistakes during their tasks, and figuring out why requires checking detailed records of their actions, which can be slow and expensive. The authors designed a new lightweight way to quickly review these task traces using smart rules and a single AI judge, focusing only on the important parts. Their approach finds problems more accurately and much faster than older methods, making it useful for real-world AI systems. It has already been used successfully in an enterprise platform.
Open → 2610.03315v1

Large language models gain flexible risk control without retraining

Risk-Conditioned Fine-Tuning of Large Language Models

Abstract: Large Language Models (LLMs) are increasingly deployed in settings where rare but severe harmful generations can have significant consequences. Existing Risk-Averse RLHF addresses this issue by optimizing Conditional Value-at-Risk (CVaR), but it trains policies for fixed risk levels and therefore cannot adjust the desired degree of risk aversion at inference time. In this paper, we propose risk-conditioned RLHF, a framework that trains a single policy that provides a continuous risk-control interface, enabling users to select different degrees of risk aversion without retraining or deploying multiple risk-specific models. Experiments across multiple benchmarks demonstrate that a single risk-conditioned policy can adapt to different risk levels at inference time, enabling more flexible and risk-aware LLM deployment.

Tue 8 SeptMachine Learning
The gist
Sometimes large language models (LLMs) can produce harmful or risky outputs that have serious consequences. The paper shows a way to teach an LLM to control how careful or risky it should be when generating text, using a single trained setup. This means users can adjust the model’s caution level anytime without needing to train different models for different risk settings. The authors tested this idea on multiple benchmarks and found it worked well, allowing more flexible and safer use of LLMs.
Open → 2609.08064v1