Place aware entropy improves click confidence in GUI agents
How Many Pixels Is a Digit Worth? Place-Aware Coordinate Entropy for GUI Agent Confidence Estimation
Artificial Intelligence
Summary
Predicting exactly where to click on a screen is tricky, and usual methods to tell if a predicted click is likely correct don’t work well. The authors found that some parts of the predicted position matter more than others, especially digits representing larger areas. They created a new way to measure confidence by giving more importance to these key parts, which helps better separate good from bad predictions. This makes GUI agents more reliable without needing extra data or expensive calculations.
What this means in practice
- •For mobile app developers: Improve reliability of automated UI testing by getting better confidence scores for predicted screen clicks without costly repeated sampling.
- •For software quality engineers: Deploy more efficient GUI automation by using place-aware entropy to quickly identify when simulated clicks are likely to be correct or incorrect.
Authors
Yunxiang Li, Xixin Wu, Helen Meng
Abstract
GUI agents predict click coordinates as digit-token sequences, but standard text-LLM confidence estimation methods rank correct clicks from wrong ones only weakly. GUI-specific alternatives use K samples or new supervision, but still leave room for improvement. We trace part of this to place-value asymmetry: bounding-box correctness often makes higher-place digits more important than lower-place digits, so uniform aggregation weakens the signal that determines correctness. The fix is to weight each digit's Shannon entropy by its place value. We call this Place-Aware Coordinate Entropy (PACE). Across fixed-scale agents on ScreenSpot-Pro and ScreenSpot-v2, PACE wins both AUROC and selective accuracy on all primary comparisons in a single forward pass, matching or outperforming K-sample baselines at a fraction of the cost. PACE provides a per-click confidence estimate that turns coordinate-token internals into a practical confidence signal for GUI agent deployment.