user interface developers

Abstract: Science fiction often features characters or objects that undergo radical changes in form while still being recognizable as the same entity. For example, a vehicle might unfold into a robot, or one body might flow or shape-shift into another. We use such transformations as a reflective lens to study similar shape-shifting in visualization, where one visual representation transforms into a completely different one. For example, a node-link diagram might transform into a parallel coordinates plot as a user's task changes from understanding relationships to identifying correlations. Based on four instances of science-fiction transformation, our paper highlights two key questions: First, why is a transformation into a target form needed, or what is the purpose? Second, what remains recognizable across the transformation from source to target, or what is the correspondence? Through four hypothetical transitions between visual representations of a multifaceted dataset, we demonstrate that purpose is also essential to visualization transformations. Furthermore, we show that the question of correspondence results in different levels of complexity in terms of designing and understanding transitions. Depending on whether correspondence is 1:1, 1:N, or N:1, visualization transformations may be direct and easy to understand, or they may require more elaborate transition constructs, implying increased cognitive effort for interpretation. Particularly, cases of 1:N and N:1 correspondence pose challenges for designing visualization transformations.

Abstract: French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model performance and prediction reliability remain insufficiently understood. To address this gap, we construct a controlled French surface-variant benchmark covering seven task types, with 4,200 original-low-high triplets (12,600 input instances). Each high-intensity variant retains all low-intensity edits and adds further changes. We evaluate six models from two LLM families. The results show that accuracy decreases and Brier score and negative log-likelihood (NLL) increase with perturbation intensity. Larger models show smaller accuracy losses under high intensity, but increases in Brier score and NLL do not consistently decrease with scale. Keyboard typos cause the largest losses in accuracy and the quality of predicted probabilities at the predefined high-intensity settings. Finally, calibration temperatures fitted on original inputs transfer effectively to surface variants. Combining calibration with input restoration further improves Brier score and NLL over either method alone while preserving restored accuracy. Our benchmark provides a unified framework for jointly evaluating task accuracy and prediction reliability under common French surface variations.

Abstract: Editing agents on 2D infinite canvases must reason about application state that pixels may not reveal. Targets may be offscreen or covered, while visual containment and touching arrows may disagree with recorded frame membership or connector bindings. We ask how a structured observation layer can support accurate object selection during agentic editing on 2D infinite canvases. We present CanvasObs, which augments visual context with a structured payload and guidance for reading four components: Identity & State, Visibility & Structure, Relations, and Offscreen Object bounds in action coordinates. A Remote Team Handoff case study illustrates retrieving an editable offscreen copy, choosing a covered chart by color, separating cards by frame membership, and preserving an evidence connection. We turn these cases into ten controlled configurations and compare CanvasObs with the baseline agent in 1,200 browser trials across three models. CanvasObs improves Correct target for every model. Combined across models, it raises Correct target from 16.5% to 48.3%, an increase of 31.8 percentage points (95% confidence interval [29.0, 34.7]), and Full task success from 4.3% to 25.0%. These results show that compact structured observation can improve object choice on 2D infinite canvases without redesigning the editing pipeline.

Abstract: We adapt Stevens's power law to measure the innate ability of AI models to read visualizations, which can reveal the built-in perceptual mechanisms of algorithmic models. In our pilot study, models see no legend. A model first views a reference visual representation and estimates its magnitude, then estimates the magnitude of each subsequent image of the same representation relative to that reference. Our evaluation of twelve visual variables makes how algorithmic models read visual encodings measurable, comparable with human perception, and more interpretable to humans.

Abstract: Opacity is a widely used channel in data visualization, but it remains less well understood compared to channels such as color, length, size, etc. Recent work from Meng et al. investigated the impact of opacity across competing color schemes, finding that certain color schemes were associated with better participant accuracy. We examine these effects further in a controlled two-alternative forced-choice setup to determine whether opacity differences are truly equal across possible opacity comparison ranges. In a within-subjects study with 96 trials, including two competing color schemes (best and worst from Meng et al.) and 48 opacity pairs, we find little differences between color schemes but larger individual differences in accuracy. Further, results show stable performance in middle opacity ranges, with more errors occurring when comparing extreme values. We discuss potential implications for design guidelines and further study and make our study materials, analysis scripts, and data available at https://osf.io/zv9dx/overview?view_only=38d03cea1b3d42788593e3e6b1016cfd.

Abstract: Browser-use agents require seamless alignment between structured web metadata and visual information, while preserving relevant context across long interactions. Existing interfaces often rely on either screenshot-level action prediction or static Set-of-Marks overlays, leaving the model to resolve dense DOM-pixel alignment before every operation. We introduce Probe to Act (P2A), an active probing framework for the browser-agent loop that moves this alignment into decision time. P2A addresses an asymmetric bridge between symbolic DOM hypotheses and screenshot layout by rendering on-demand symbolic DOM structure back into pixels. Before committing a state-changing browser operation, the agent can issue lightweight probes to translate DOM handles into pixel evidence, map screen regions back to DOM candidates, register visual-only targets, and commit verified notes. These interleaved processes naturally produce evidence-based memory: only probed, acted-on, or explicitly committed observations are kept across steps, preserving only decision-critical evidence in long-horizon contexts. P2A can be used as a prompting strategy for proprietary models under the standard DOM+SoM interface, and can be distilled into open-weight models through cold-start synthesis and self-bootstrapped SFT. Across three browser-use benchmarks, P2A shows clear gains on task success rate for both proprietary and fine-tuned models; on VisualWebArena, for example, it improves Gemini-3-Pro from 54.1% to 61.2% and Qwen3-VL-8B from 24.6% to 32.9%, while matching the costly full-observation history ($\sim$3$\times$) at only $\sim$1.2$\times$ the peak retained input context of action-only history.