Papers for

web interface designers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Proactive verification improves reward reliability for gui agents

VeriCUA: Learning Proactive Verification for Reliable GUI Agent Rewards

Abstract: Reinforcement learning for computer-use agents increasingly relies on proactive reward agents that interact with the post-execution GUI environment to verify what recorded screenshots cannot reveal. The reliability of such verification hinges on two coupled decisions: what to check and when to stop. Existing methods leave these decisions to prompted foundation models. However, neither scaling up the model nor providing few-shot examples reliably improves verification: prompted Judges ranging from 4B to GPT-5.4 all plateau at 68%-70% macro-average agreement. This is because the value of a check depends on the environment rather than the task alone, and becomes apparent only through execution. Learning from execution is itself constrained, as the Actor's terminal state cannot be faithfully replicated during training, allowing only one verification attempt per state. We introduce VeriCUA, a proactive reward agent that learns its verification strategy from these single attempts. A trainable Judge decides what to check and when to stop, while a frozen Probe executes the checks in the live environment. Online reinforcement learning optimizes the Judge for both verdict correctness and efficient Probe use, with returns normalized across terminal states. Trained only on OSWorld and evaluated without further training on MobileWorld and WebArena-Verified, VeriCUA achieves the highest reward agreement across all three benchmarks and lies on the agreement-cost Pareto frontier among proactive verifiers. It surpasses few-shot GPT-5.4 by 5.36 points with 47.6%-59.8% fewer Probe tokens, and even a 1B VeriCUA exceeds every prompted Judge. As the reward for Actor RL, it raises task success by 3.21 points overall and 5.40 points on held-out multi-application tasks. These results identify the verification strategy, not verifier scale, as the key to reliable GUI rewards.

Mon 19 OctArtificial IntelligenceMachine Learning
The gist
Reinforcement learning agents that use graphical user interfaces (GUIs) need rewards that show if they completed tasks correctly. Current methods verify outcomes by asking large language models what to check and when to stop, but they don’t get much better with bigger models. The authors introduce VeriCUA, which learns what to check and when to stop by training a component called the Judge, using only one chance to verify each state. VeriCUA performs better than existing large language models on multiple benchmarks, improving both reward reliability and task success.
Open → 2610.26469v1

Benchmark measures precise image edits on structured graphics

Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets

Abstract: Fine-grained image editing requires more than producing a visually plausible result: an editor must execute the requested attribute change precisely while leaving everything else intact. However, existing benchmarks leave a critical gap between realism and verifiability: benchmarks built on realistic images typically rely on human or vision--language model judgments, while deterministic evaluation has largely focused on synthetic shape canvases, with application-oriented extensions primarily limited to charts. This makes it difficult to determine precisely how much of a requested edit was executed, where unintended changes occurred, and whether small differences between models reflect genuine editing capability or evaluator uncertainty. To bridge this gap, we present VeriEdit-Bench, a benchmark for fine-grained, instruction-faithful image editing across realistic structured assets with deterministic, four-axis evaluation. Its 1,740 cases are compiled from the source code of 153 Scalable Vector Graphics (SVG) graphics, charts, web interfaces, and presentation slides. Controlled source-code edits preserve the original visual context while yielding exact target images, pixel-level edit masks, and explicit edit specifications, enabling reproducible scoring along four axes: edit fidelity, preservation, localization, and magnitude. Evaluating eleven editors, we find that even the strongest model remains far from full credit; rankings for the same recoloring operation reverse between charts and SVG graphics; and outputs with similar pixel-accuracy profiles can still differ substantially in localization and change magnitude. This decomposition yields graded, verifiable feedback and exposes model-specific capability and failure profiles that holistic scores or evaluator-dependent judgments may obscure.

Sat 3 OctComputer Vision and Pattern Recognition
The gist
Fine-grained image editing means changing exactly what you want in a picture without messing up the rest. The authors found that existing tests either focus on realistic images with unclear scoring or on simple synthetic images that don’t fully match real uses. They created VeriEdit-Bench, a new test collection using detailed edited vector images where changes are exactly known. This lets them measure how well image editors follow instructions in multiple ways, showing that even the best models still make noticeable mistakes.
Open → 2610.04381v1