banking operations teams

Abstract: We explore the ability of humans to compare strings, an error-prone activity which occurs in critical contexts (e.g., bank wire transfers), where it is not always possible to trust a machine to help. We hypothesize that understanding human cognitive and perceptual limitations can improve how people verify whether two character strings are identical. We vary the strings' relative positions and presentation formats to measure comparison speed and accuracy. We present results from two online experiments with 178 human subjects who compared pairs of strings to determine whether the two strings were identical. The strings varied by length (5 characters to small paragraphs), difference (matching versus making a random change or a hard-to-detect one), alignment (vertical versus horizontal), and presentation (Overlay versus Adjacent). Our experiments test whether, by encoding and memory tasks into perceptual tasks, overlay formats can offer advantages in speed and accuracy. Results show that overlay offers a substantially faster method, with improved accuracy for long strings. Nevertheless, mean error rates were at least 4 percent, indicating that even the improved approach may be insufficiently accurate for many verification tasks.

Abstract: Frontier language models now produce professional deliverables that expert graders judge to match human work on a substantial share of economically valuable tasks, yet most enterprise GenAI initiatives fail to show a measurable business effect and a large fraction of agentic projects are expected to be cancelled. We argue that this is substantially a measurement problem: public benchmarks answer "what can the model do?", whereas a deployment decision requires "is this workflow fit, reliable, safe and worth scaling - here, on our data, under our controls?". We present EnterpriseVal, a use-case-level evaluation system that closes this gap. It comprises (i) a formal specification of the use case and of the frozen socio-technical configuration under test, model, prompts, retrieval, tools, guardrails and human oversight, with an autonomy level and consequence tier that jointly set the required evaluation intensity; (ii) a metric catalogue spanning fidelity, utility, efficiency, reliability, assurance and oversight; (iii) a grading protocol that scales blinded expert judgement with calibrated LLM-as-judge scoring through prediction-powered inference; (iv) a two-tier threshold gate, stated as an executable algorithm, that maps metric vectors with confidence bounds to REJECT/CONDITIONAL/SCALE decisions; and (v) a value-and-risk model in which the reviewer catch rate is a measured parameter. We report a pilot across three workflows in a global bank. In credit-memo drafting, human-graded citation precision reached 88% and hallucination rate 1.6% for the best model against gates of 70% and 5%; in procedure transformation, analyst refinement effort fell from an estimated 27.4 to 2.9 hours per document. We separate established results, documented pilot evidence, the proposed system and open hypotheses, and specify the experiments required for full validation