Papers for

ai tool integrators

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Japanese financial exams power new updatable language model test set

JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams

Abstract: Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.

Fri 13 NovArtificial Intelligence
The gist
Language models like ChatGPT need special tests to see how well they understand finance in different countries. Existing tests often don’t fit specific countries or get outdated when their questions leak online. The authors made a new test set called JFinBench from recent Japanese finance exams that keeps updating. They created a process where a computer program helps format exam questions but a person checks and corrects everything, making updates easier. Testing several language models showed some perform very well, especially when they can also search the web or use a terminal, but the model’s own knowledge still matters.
Open → 2611.18130v1

Large language models benchmarked on high performance computing code optimizations

HPC-Bench: A Comprehensive Benchmark for High Performance Computing Codes

Abstract: Large Language Models (LLMs) have shown strong capabilities in code generation and reasoning, motivating extensive benchmarks for evaluating functional correctness. However, performance optimization in High-Performance Computing (HPC) poses fundamentally different challenges that are not captured by existing evaluations: HPC performance depends on low-level code, intricate memory access patterns, and computational kernels embedded within sophisticated execution contexts. We introduce HPC-Bench, a performance-grounded benchmark for evaluating LLMs as optimizers of computational kernels in realistic HPC workloads, including 114 workloads across 16 HPC computational motifs. HPC-Bench provides a fully automated, end-to-end evaluation pipeline that combines program-level correctness verification with two correctness-gated metrics, Fast@k and Speedup@k, to characterize reliability and speedup under a fixed sampling budget. Our evaluation of three representative LLMs across serial CPU, OpenMP, and CUDA settings shows that stricter speedup requirements substantially reduce the probability of finding a correct and fast optimization. HPC-Bench is available at https://github.com/Deep-Learning-Profiling-Tools/HPC-Bench.

Mon 2 NovPerformance
The gist
Making computer programs run faster in high-performance computing is tricky because it depends on detailed, low-level code and how data moves around in memory. The authors created HPC-Bench, a new test that measures how well large language AI models can improve these computing codes across many real-world tasks. The test checks both if the optimized code still works correctly and if it runs faster. They showed that finding optimizations that are both correct and much faster is difficult for today’s AI models.
Open → 2611.02752v1

Pairing coding agents and tools greatly affects their success and efficiency

Pairing Matters: Performance, Failures, and Efficiency in Lightweight Coding Agents

Abstract: Self-hosting coding agents is often necessary for software-engineering teams that must keep proprietary code and development traces within controlled infrastructure. Lightweight, open-weight models make such deployment more feasible by reducing hardware requirements, while strong results on repository-level benchmarks demonstrate their potential for demanding coding tasks. Deploying these models as coding agents requires pairing them with an agent harness, making their joint performance a key deployment consideration. Yet existing leaderboards primarily report fixed harness-model configurations rather than controlled comparisons across alternative pairings. To address this gap, we investigate harness-model interactions through a controlled full-factorial evaluation of six open-source harnesses and three lightweight coding models across 100 SWE-bench Verified tasks, producing 1,800 graded runs under common serving, metering, and grading conditions. For the same model, resolution rates differ by up to 44 percentage points across harnesses, while harness rankings change across models. We apply an evidence-based failure taxonomy to 1,100 unsuccessful runs and find substantial differences in failure patterns across harness-model pairings. A model's dominant failure modes under one harness need not dominate under another. Efficiency is similarly dependent on the pairing: even among model-directed loop harnesses, charged tokens per resolved task vary by more than 20-fold. These findings make harness-model selection a central deployment decision for lightweight coding agents. Strong performance with one harness or model does not guarantee similar results with another. We translate these findings into practitioner guidance for pairing selection, token-efficiency assessment, and trajectory-informed harness engineering, and release the complete experimental records and analysis artifacts.

Fri 23 OctSoftware Engineering
The gist
It matters a lot how lightweight coding AI models are paired with the software tools (called harnesses) that run them. The authors show that one model can perform very differently depending on which harness it uses. They tested many combinations and found that failure reasons and efficiency vary widely. This means picking the right harness and model pair is very important to get the best results when deploying coding AI agents.
Open → 2610.32131v1

SideKernel improves sandbox usability for AI coding agents on macOS

SideKernel: A Usable microVM Sandbox for AI Coding Agents on macOS

Abstract: AI coding agents are untrusted system components, yet they require autonomy on the developer machines they run on. This contradiction is a security problem. Sandboxes provide an isolated environment, but for local macOS development, the existing local, open-source options for AI coding agents are few in number and cumbersome to use. I conducted a formative online user survey which indicates that fewer than 40% of AI coding agent users run their agents in a sandbox and identifies the top usability barriers hindering AI coding agent sandbox adoption. These findings are used to develop SideKernel: an open-source, local, microVM-based macOS sandbox for AI coding agents designed for usability. To evaluate SideKernel, I compiled a list of sandboxes available on the market and filtered it against five inclusion criteria. Then I performed a comparative analysis between SideKernel and the sandboxes that satisfy these criteria, across 23 capability tests derived from the usability barriers revealed by the user survey. I discovered that only a few sandboxes are similar to SideKernel, and that among those, Docker Sandboxes and SideKernel score highest on capability features related to usability. A secondary contribution of this paper is a survey of the existing solution space for local, open-source, microVM-based macOS sandboxes for AI coding agents.

Thu 1 OctCryptography and SecurityArtificial Intelligence
The gist
Running AI coding assistants on personal Macs can be risky because these tools need a lot of freedom but might do harmful things. The authors found that less than 40% of users put these assistants in a safe, separate space called a sandbox, mostly because current options are hard to use. They built SideKernel, a simple and open sandbox solution that works well on Macs. SideKernel performs strongly compared to other similar sandboxes, making it easier for developers to safely run AI coding helpers.
Open → 2610.02456v1

Common problems and fixes in open-source AI multi-agent systems

Understanding Issues, Causes and Solutions in Open-Source LLM-based Multi-Agent Systems

Abstract: With the advancement of LLM-based multi-agent systems (MAS), an increasing number of opensource projects are adopting multi-agent architectures as the foundation of their core functionality. Although research and practice on MAS have attracted considerable attention, limited studies have explored the challenges faced by practitioners of open-source LLM-based MAS, the causes of these challenges, and potential solutions. To address this gap,we conducted an empirical study to understand the issues that practitioners encounter when developing and using open-source LLM-based MAS, the possible causes of these issues, and potential solutions. We collected 22,848 closed issues from 21 open-source LLM-basedMASand applied a mixed automated and manual filtering approach to reduce the dataset to 944 issues related to LLM-based MAS.We then analyzed these issues to understand the frequent issues encountered by practitioners, their underlying causes, and potential solutions. Our study results show that (1) Orchestration & Execution Issue is the most common issue faced by practitioners, (2) Workflow Problem, Tool Integration Problem, and Memory Problem are identified as the most frequent causes of the issues, and (3) Optimize Workflow is the predominant solution to the issues. Based on the study results, we derive empirically grounded implications for practitioners and researchers aimed at improving orchestration, tool integration, and memory mechanisms in LLM-based MAS.

Thu 1 OctSoftware EngineeringArtificial Intelligence
The gist
Open-source systems where many AI agents work together often face challenges. The study looked at nearly 23,000 problem reports to find common issues, causes, and solutions. They found that managing how these agents work together (orchestration) is the biggest problem. Problems with workflow, connecting different tools, and memory were common causes. Fixing workflows was the main way to solve these issues.
Open → 2610.00905v1

Benchmark reveals varied strengths of language models on quantum tasks

QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

Abstract: We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation. Across 2,750 evaluations (10 models $\times$ 11 tasks x 5 difficulty levels x 5 seeds), we find that overall rankings mask substantial per-task variation. The Spearman correlation between overall and per-task rankings is statistically insignificant for 4 out of the 11 tasks included in this benchmark. A 2-parameter Item Response Theory (IRT) model validates measurement quality, and prompt sensitivity analysis confirms ranking robustness across prompt conditions. All tasks are auto-verifiable via execution, thus not requiring any manual evaluation. We make the code and data publicly available on Huggingface.

Mon 28 SeptArtificial IntelligenceMachine Learning
The gist
Evaluating how well large language models understand quantum computing, the authors created a test covering many different quantum tasks like building circuits and fixing errors. They found that a model that does well overall might not do well on some specific tasks, showing that these skills vary a lot. The benchmark runs tasks automatically, so no manual checking is needed, and their methods ensure reliable results. They also shared the test and data openly for others to use.
Open → 2609.35581v1

Benchmark evaluates AI email agents on enterprise productivity tasks

EmailBench: A Benchmark for Evaluating LLM Agents on Enterprise Email and Productivity Tasks

Abstract: Enterprise email agents must combine information retrieval, structured state changes, temporal reasoning, and multi-step coordination. Recent agent benchmarks include productivity tasks, but few center on typed email workflows in a self-contained environment. We introduce EmailBench, a benchmark of 206 email and productivity scenarios across 16 task categories. The benchmark couples a typed email API specification with provider-neutral naming, a deterministic synthetic Enron-inspired corpus, and a scenario suite whose topic selection was informed by aggregate task-intent telemetry from an interactive prototype. Its hybrid evaluation protocol combines 258 executable static assertions with 211 LLM rubrics. We evaluate eight LM configurations on a fixed single-user corpus. The best-performing configuration passes only 33.5% of scenarios despite 99.7% of its tool calls completing without an observed API failure, with pass rates varying substantially across task categories. This gap shows that valid tool execution is not equivalent to task completion. EmailBench provides a self-contained environment for end-to-end email-agent evaluation, with broader tool coverage, multi-persona testing, and repeated-run evaluation as future work areas.

Fri 25 SeptArtificial Intelligence
The gist
Handling enterprise email involves tasks like retrieving information, managing schedules, and coordinating multiple steps accurately. The authors created EmailBench, a set of 206 email-related tasks designed to test how well AI agents can perform these real-world workflows using a synthetic email dataset. They tested eight AI configurations and found that even the best one completed only about a third of tasks correctly, showing that just completing API actions doesn’t mean the task was truly done. This benchmark helps developers measure and improve AI email assistants in a controlled, realistic setting.
Open → 2609.31906v1

Interactive authoring tools improve science data visualization and analysis

"Here Be Sharks!": Enhancing Scientific Communication and Analysis through Authoring Interactivity

Abstract: We report on a case study for designing authoring environments for interactive visualizations to enhance scientific work. We conducted a workshop and prototype review with a group of marine biologists. When it came to visualizing their data, participants identified challenges in conveying their research accurately and completely as well as and analyzing it with ease. Based on our findings, authoring environments should consider the context of scientific work aspects such as domain expertise, collaboration culture, and publication traditions. We propose consideration of non-traditional programming languages and environments for scientific work, discuss ways of facilitating interactive visualizations for scientists, and examine ways of meaningfully integrating AI. The critiques, artifacts, and reactions from the scientists along with our analysis and discussion inform how computational tools should be designed for scientific work.

Tue 8 SeptHuman-Computer Interaction
The gist
Scientists often find it hard to show their data clearly and analyze it easily. This paper looks at how creating interactive tools for writing and showing data can help scientists, especially marine biologists. The authors found that tools should fit the scientists’ specific ways of working and sharing results. They also suggest using new kinds of programming and adding artificial intelligence to make these tools better.
Open → 2609.08386v1