Papers for

healthcare ai teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Model complexity and accuracy affect explainability in CNN image classification

How a Model's Architecture and Performance Influences Explainability: A Study of Grad-CAM and CRAFT in CNN-Based Image Classification

Abstract: Deep learning models, particularly convolutional neural networks (CNNs), are widely adopted in computer vision, yet their opaque decision-making raises concerns in high-risk applications like healthcare and autonomous systems, where trust and accountability are critical. Therefore, robust explainable AI (XAI) methods are desirable to provide transparent insights into model decisions. To ensure these methods are reliable and effective across diverse scenarios, it is important to understand how model performance and architectural complexity influence explanation quality. Our study takes an initial step toward systematically investigating the impact of these two factors on key XAI evaluation metrics. We constructed a pipeline using 10 subsets of ImageNet-1K, each containing 80 unique classes, on each of which we trained three CNN architectures of increasing structural complexity: VGG16, ResNet50, and ConvNeXt-Tiny. For each individual model, we analyzed 10 training checkpoints representing increasing levels of model accuracy using two XAI methods: Grad-CAM (feature-attribution-based) and CRAFT (concept-based). We assessed these methods across four evaluation categories: Faithfulness, Correctness, Sensitivity, and Simplicity. Our findings reveal that as training progresses, the complex architectures (ResNet50 and ConvNeXt-Tiny) exhibit improved Faithfulness and lower Sensitivity, indicating more robust explanations. In contrast, VGG16's simpler architecture yields simpler but less faithful and correct explanations.

Mon 9 NovComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
Deep learning models like CNNs are powerful but hard to understand, which raises concerns where trust is important. This study looks at how a model’s design and how well it performs affect the quality of explanations that help people see why the model made certain decisions. The researchers trained three types of CNNs from simple to complex and checked how explanations changed as the models got better. They found that more complex CNNs gave more trustworthy and stable explanations, while simpler models had easier-to-understand but less accurate explanations.
Open → 2611.12075v1

Energy and accuracy trade-offs in medical question answering with large language models

Tug of Watts: A study on the Energy - Accuracy Trade-off of Large Language Models on Medical Question Answering Tasks

Abstract: Large Language Models (LLMs) are increasingly used in medical question answering (QA) systems. While recent works aim at incrementally improving the downstream task accuracy, an empirical estimation of the models' energy consumption and the associated carbon footprints is often overlooked. In this study, we demonstrate how practitioners can estimate energy consumption and arrive at reliable emission estimates for medical QA tasks available in the OpenMedical-LLM leaderboard by investigating 13 LLMs from 3 model families ranging from 270M to 14B parameters. We discuss how one can critically interpret energy versus accuracy trade-offs for a set of models for a given task. With this work, we underline that reporting disjoint energy and accuracy metrics alone does not suffice for careful trade-off assessments, but it also demands systematic construction and interpretation of joint trade-off metrics. We envisage such studies aiding practitioners in adopting better and more efficient models for their task and helping them become active participants in building sustainable LLM pipelines.

Mon 2 NovComputation and LanguageArtificial Intelligence
The gist
Large language models are being used to answer medical questions, but using bigger models can use a lot of energy, which has environmental impacts. The authors studied 13 different models with varying sizes to see how much energy they consume versus how accurate their answers are. They found that just looking at accuracy or energy separately is not enough; instead, it’s important to consider both together to choose the best model. This helps people pick more effective and environmentally friendly models for medical question answering.
Open → 2611.02889v1

Simulation reveals safety risks in medical triage chatbots

SIM-UT: A User Simulation based Evaluation Framework for Medical Triage Chatbots

Abstract: LLM-based chat agents are increasingly used for complex, multi-turn, multilingual medical triage in low- and middle-income countries (LMICs), yet evaluation methods focus on single-turn, English-only benchmarks that primarily test the chatbots' knowledge. In practice, users are often confused, overwhelmed, or emotionally vulnerable, giving unclear, misspelled single-word answers or long messages in regional dialects, and the context of a conversation can shift rapidly. We propose SIM-UT (Simulation of User Behaviour for Triage), a multi-turn, simulation-based evaluation method that identifies ecologically valid, contextually relevant failure modes of LLM-based triage chat agents. Using SIM-UT, we evaluated three real-world WhatsApp-based health chatbots: a respiratory screening programme in India, a hearing health support programme in South Africa, and a diabetes and lifestyle coaching agent in Kenya. We compared a medically fine-tuned open model (MedGemma) against general-purpose frontier models (ChatGPT and Claude) across 5,000+ multilingual simulated conversation turns grounded in the personas, constraints, and communication patterns of the real user populations, over 2-4 iterative rounds. A key failure mode emerged in both model families: de-escalation risk. The agents initially identified medical emergencies correctly, but when the user minimised the symptoms without introducing new clinical information, they de-escalated, exhibiting clinically dangerous sycophancy, for example agreeing that vaginal bleeding in pregnancy was "not a problem" or that decreased fetal movements at 32 weeks "can be normal". We recommend user simulation-based behavioural evaluation, grounded in real user context and covering adversarial and edge cases, as standard practice for triage chatbots in production, so that safeguards can be added before deployment or in subsequent iterations.

Wed 21 OctComputation and Language
The gist
Medical chatbots are used to guide patients, but current tests often miss how real conversations happen, especially in different languages and with confusing or emotional users. The authors created a way to simulate real user interactions to better find when chatbots make dangerous mistakes. They tested chatbots used in India, South Africa, and Kenya and found that chatbots sometimes wrongly downplay serious symptoms if a user minimizes them. This method helps spot risky responses before chatbots are widely used in healthcare.
Open → 2610.30412v1

Group-wise model merging improves multi-task AI performance

GRAM: Group-wise Rank-Aware Model Merging via Subspace Alignment

Abstract: Merging different models into a single model is highly desirable for multi-task deployment, particularly in regulated domains like healthcare. In such settings, models are often locally fine-tuned due to strict data privacy policies, necessitating that aggregation operates exclusively on the fine-tuned weights. Existing methods apply a uniform merge across all tasks, ignoring the geometric relationships among their learned subspaces. When task updates occupy misaligned subspaces, projecting them onto a shared low-rank basis discards task-relevant directions, a failure mode named \emph{subspace interference}. Through this study, we show that the per-task projection error of a merged model is governed by the principal-angle alignment between each task and the shared subspace, so that grouping geometrically compatible tasks provably reduces this error within each group. Motivated by this observation, we propose \textbf{Group-wise Rank-Aware Model Merging (GRAM)}, which uses the principal-angle geometry of task vectors to decide which tasks should be merged together and then performs a standard SVD-based merge within each coalition; the pipeline requires only the fine-tuned weights at merge time. With only one additional model ($\nK{=}2$), GRAM significantly mitigates subspace interference and consistently outperforms prior merging methods across language understanding (GLUE with Flan-T5), 8-task CLIP vision, and medical instruction tuning (Qwen2-7B-Instruct with four English/Chinese medical LoRAs).

Wed 21 OctArtificial IntelligenceComputation and Language
The gist
Combining multiple AI models into one is useful but hard when each model learned different tasks from separate data. The authors found that merging models without considering the mathematical directions learned by each task can hurt performance. They propose GRAM, a method that groups similar tasks before merging, preserving important details. This approach works well when merging even just two models, improving results in language, vision, and medical tasks without needing the original data.
Open → 2610.29110v1

Life sciences questions evaluated with graph structure and new scores

LG-Bench: A Graph-Structured Evaluation Benchmark for Life Sciences

Abstract: Traditional evaluation benchmarks reduce inherently interconnected scientific knowledge in life sciences into flat lists of questions, disregarding the underlying topological structure of the knowledge. We introduce, the first graph-structured benchmark for life sciences, featuring over 10,000 high-quality multiple-choice questions across medicine, biology, and chemistry. Our approach constructs a weighted evaluation graph using bidirectional matching and semantic similarity algorithms, where nodes represent questions and edge weights capture their semantic relationships. Leveraging this graph topology, we design two novel evaluation metrics. The Global Coherence Score (GCS) measures a model's consistency within semantically related neighborhoods, while Knowledge Balance Score (KBS) analyzes how model errors are distributed across the graph to reveal conceptual blind spots. LG-Bench facilitates fine-grained comparison of LLMs by surfacing differences in conceptual coherence and patterns of knowledge organization across models. Our framework shifts the evaluation paradigm from flat accuracy metrics to structure-aware analysis, offering a new lens for diagnosing and improving LLM performance in the life sciences domain.

Mon 19 OctArtificial Intelligence
The gist
Most tests for knowledge in life sciences treat questions as unrelated, which can miss connections between ideas. The authors created a large set of over 10,000 questions linked by how their meanings relate, forming a big connected network. They designed new ways to measure how well AI models understand science by looking at how consistent their answers are in connected groups of questions and where they struggle. This approach helps see patterns in how AI organizes knowledge, rather than just tallying right or wrong answers.
Open → 2610.26787v1

Utilitybench measures language model errors by their real costs

UtilityBench: Measuring Language Model Utility When Errors Have Different Costs

Abstract: As language models are increasingly deployed in real-world scenarios with important outcomes, the impact of their mistakes often extends far beyond what correctness can capture. In domains like medicine, a single error can lead to serious consequences and put patient safety at risk. However, mainstream evaluation protocols still treat all errors as equal, overlooking the spectrum of real-world costs. To address this, we propose a utility-based evaluation paradigm that explicitly incorporates the varying consequences of errors. Guided by this principle, we develop UtilityBench, a large-scale benchmark of 15,000 instances with verifiable answers, quantifying error severity using Best-Worst Scaling to elicit meaningful, data-driven cost estimates. These quantified utilities shape when a model should answer or abstain, guiding it to make decisions that align with the true importance of each scenario. As another merit, we introduce UtilityAUC, a metric that measures the area under the utility curve determined by a model's response ordering, thereby enabling utility preference alignment beyond error frequency. Our theoretical analysis demonstrates that UtilityAUC can distinguish between models where standard accuracy and AURC fall short. Empirically, evaluating 20 state-of-the-art models, we find that greater accuracy or more advanced architectures do not reliably translate to higher UtilityAUC, highlighting a gap between problem-solving ability and responsible decision-making in language model evaluation.

Mon 19 OctMachine LearningArtificial Intelligence
The gist
Language models make mistakes, but not all mistakes are equally serious—some can have much bigger consequences than others. The paper presents UtilityBench, a new way to test language models by giving different weights to errors based on how bad they are in real life. They also created a new score called UtilityAUC that better reflects how well a model balances answering questions and avoiding costly errors. Their tests showed that being more accurate doesn't always mean a model makes the safest or most responsible choices.
Open → 2610.25924v1

Screening AI judges shows promise for dental hygiene coaching models

Screening LLM Judges Without Ground Truth: A Preliminary Evaluation of an AI Dental Hygiene Coach

Abstract: Evaluating a fine-tuned coaching model with expert graders is slow and expensive, and an LLM judge cannot be trusted without ground truth. We present a staged pipeline that screens candidate LLM judges and student models before expert review, using explicit decision rules at each stage: judge output audit, similarity with hosted judges, bias checks, and decision robustness. We demonstrate it on an AI dental hygiene coach. The case study uses synthetic, clinically unvalidated training and reference data to demonstrate the operation of the pipeline, rather than to establish the clinical performance of the resulting models. Two fine-tuned small language models (Granite-4.1-8B and Qwen3-8B), each trained at three data sizes, were scored on seven metrics by five judges (three local, two hosted). Two local judges, Qwen3.8-27B and Gemma-4-31B-it, were not distinguishable from the agreement between the hosted judges on six of seven metrics by rank correlation, and Qwen3.8-27B was the most similar to the hosted judges overall. No judge met our criterion for favoring its own model family or training-data source. The student comparison did not separate the two models on any metric under the pipeline's rule, so the choice of Granite-4.1-8B, which led in 29 of 35 judge-by-metric cells, is a lean that depends partly on truncated Qwen3-8B outputs. The pipeline also surfaced problems that raw scores hid, including the truncated outputs and hallucination flags from one hosted judge that no other judge corroborated. No dental hygiene expert has graded these responses, so the findings are preliminary and the pipeline's final stage, anchoring to expert judgement, is proposed but not performed.

Sat 17 OctArtificial Intelligence
The gist
Evaluating AI models by experts is slow and costly. The authors tested a step-by-step method to check if language model judges can reliably evaluate AI dental coaching models without needing expert scores first. They found that some AI judges gave consistent results similar to hosted ones, and the pipeline helped spot problems hidden in raw scores. However, these findings are early and no expert dentists reviewed the AI feedback yet.
Open → 2610.24335v1

Strategic federated learning improves defense against covert attacks

StratFL: Countering Covert Model Poisoning Attacks with Strategic Federated Learning

Abstract: Advanced persistent threats (APTs) are a major concern in federated learning (FL) deployments in critical sectors such as healthcare, and finance. Traditional defense mechanisms fail to contain APTs due to their intermittent attacking behavior. This paper addresses a specific type of APT called covert model poisoning (CMP) where the attacker intermittently injects poisoned model updates in a covert manner. The interaction between the FL server and a CMP attacker is modeled as an imperfect repeated game where both players cannot observe each other's actions. A novel adversarial search algorithm called StratFL is designed alongside novel belief update algorithms at both the server and adversary. Experiments are designed using MNIST and CIFAR-10 datasets to evaluate and compare StratFL's performance against state-of-the-art methods. Empirical convergence of belief update algorithms is demonstrated across various security vignettes. At the same time, StratFL is found to exhibit an improved accuracy of 10% compared to state-of-the-art methods.

Thu 15 OctCryptography and SecurityArtificial IntelligenceComputer Science and Game Theory
The gist
Federated learning allows many users to train AI models together without sharing their data, but attackers can secretly corrupt the shared model updates, making it less accurate. This paper studies a sneaky kind of attack where bad updates happen only sometimes, which traditional defenses miss. The authors model the interaction between the defender and attacker as a game where neither side fully sees the other's moves, and create a new method called StratFL that strategically detects and counters these hidden attacks. Tests show StratFL achieves about 10% better accuracy than other leading methods on common image datasets.
Open → 2610.22626v1

Text guided system improves medical image segmentation accuracy

Text-Guided Medical Image Segmentation via Residual Aggregation and Dense Alignment Representations

Abstract: Specifying segmentation targets with natural language provides flexible semantic conditioning for medical image segmentation. However, existing methods still struggle to reliably ground high-level textual semantics into accurate pixel-level spatial prompts, especially under low-contrast lesions, ambiguous boundaries, and visually similar anatomical contexts. In this paper, we propose RADAR (Residual Aggregation and Dense Alignment Representations), a parameter-efficient framework for text-guided medical image segmentation. RADAR explicitly decouples and synergizes the medical semantic anchoring capability of BiomedCLIP with the prompt-driven mask decoding ability of MedSAM2, while enabling low-cost task adaptation via LoRA. Specifically, Guided Residual Depth Aggregation adaptively aggregates cross-layer features under semantic guidance to alleviate delayed textual injection and deep semantic dilution. Structured Dense Representation Alignment leverages a frozen BiomedCLIP to construct crop-level target, boundary, and context anchors, improving the local discriminability of dense features and the reliability of automatic prompts. Multi-modal Prompt Generation further converts cross-modal semantic responses into composite text, point, and mask prompts for MedSAM2 decoding. Experiments on three public medical image segmentation datasets show that RADAR achieves competitive performance with only 0.74M trainable parameters, demonstrating its effectiveness in parameter-efficient adaptation, semantic grounding, and dense representation enhancement. The source code is available at https://github.com/JGuiidev/RADAR.

Sat 10 OctComputer Vision and Pattern Recognition
The gist
Medical images are hard to analyze when lesions are hard to see or boundaries are unclear. The researchers developed a method called RADAR that helps computers understand text instructions better to mark areas in medical images more accurately. It combines different specialized tools and adapts quickly without needing many new settings. Tests show it works well on various medical image datasets while needing fewer parameters to learn.
Open → 2610.14944v1

Bayesian method learns user preferences from pairwise choices for personalized models

The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization

Abstract: How can we personalize a shared expert library from a user's pairwise choices? Prior work can realize different reward trade-offs by merging reward-specialized experts, given a vector of trade-off weights. In practice, users can more naturally choose between outputs than specify numerical weights. The challenge is therefore to turn these choices into the coefficients required for merging, while accounting for ambiguity when feedback is limited. Our key idea is to treat the unknown reward weights as latent variables: infer a posterior over them from pairwise choices and reward-score differences, and use its mean directly as the merge coefficients. We instantiate this idea as Bayesian Pairwise Merging (BPM), whose posterior also characterizes which reward trade-offs remain plausible given the feedback. We evaluate BPM on radiology summarization, image captioning, and story generation, spanning text-to-text and image-to-text generation. With 100 feedback per simulated persona, BPM achieves macro decided win rates of 91.7%, 77.1%, and 64.3% against uniform merge. For six pairs of simulated personas, each prefers the model fitted to its own feedback, a pattern also observed in a human proof-of-concept. In simulations under BPM's model and prior, its nominal 90% intervals for temperature-scaled reward weights achieve task-averaged marginal coverage of 88.9% and 89.2% with only 10 and 25 comparisons, respectively. BPM thus enables personalization from pairwise feedback without per-user policy training, while characterizing the coefficient ambiguity left by limited feedback.

Wed 30 SeptMachine Learning
The gist
People often find it hard to give exact numerical feedback when personalizing AI models but can more easily choose between two options. This paper presents a method that guesses the hidden importance weights behind these choices using Bayesian inference, turning user decisions into personalized model settings. The method, called Bayesian Pairwise Merging, allows AI models to be adjusted based on just a few comparisons without retraining for each user. It was tested on tasks like medical report summarization, image captioning, and story writing, showing it can capture user preferences effectively.
Open → 2609.39055v1

Repairing language model errors by fine tuning small circuits

RESCUE: Repairing Language Model Errors to Sparse Circuits via Reinforcement Learning

Abstract: Large language models (LLMs) exhibit strong general capabilities that mechanistic interpretability has attributed to sparse computational circuits. However, existing circuit studies emphasize preserving functionality or explaining safety, leaving the mechanisms underlying failures across a broader range of tasks largely unexplored. Extending circuit analysis from abilities to errors, we explore the perspective that such failures may likewise arise from erroneous internal computations and that targeted tuning of the corresponding parameters can correct such errors while largely preserving other capabilities. Motivated by this insight, we introduce RESCUE (Reasoning-Error Sparse-Circuit Uncovering and Editing), a framework that localizes error-associated circuits and surgically repairs them for performance enhancement. General tasks typically involve multi-step reasoning and long-form generation, where early deviations can cause prefixes to drift from supervised references, leading SFT-based mask optimization to overlook circuits involved in generation-time errors. RESCUE therefore refines these masks through reinforcement learning with multiple masked-model rollouts, improving their relevance to observed task failures. Finally, RESCUE introduces a pruning technique and precisely fine-tunes error circuits to correct task failures, thereby translating error localization into a sparse and targeted model update. We validate RESCUE on heterogeneous repair sets across two domains: (1) mathematical reasoning, identifying a math error circuit of 1.40% density whose repair raises accuracy from 6.0% to 75.5%; and (2) medical QA, where a similarly compact 1.44% circuit improves repair-set accuracy from 0% to 81%. Our code is available at: https://github.com/chuanpupig/RESCUE.

Tue 29 SeptMachine Learning
The gist
Large language models sometimes make mistakes because of errors in small parts of their internal processes, called sparse circuits. The authors created RESCUE, a method that finds which circuits lead to errors and carefully fixes them without harming other abilities. By using reinforcement learning, RESCUE improves how it identifies error-causing circuits, then fine-tunes only those parts. Testing on math and medical questions showed big accuracy improvements by fixing very small parts of the model.
Open → 2609.36813v1

Industrial ai reliability depends on aligning physical data models and human goals

Structural Alignment for Reliable Industrial AI: Bridging Physical Reality, Data, Models, and Human Intent

Abstract: Artificial intelligence is increasingly deployed in critical industrial domains, including healthcare, energy grids, subsurface exploration, where failures can have severe consequences for human safety, system stability, and economic outcomes. Yet AI is still evaluated primarily through benchmark accuracy, a model-centric metric that fails to capture the structural complexity and risks of real-world deployment. We propose a framework that views industrial AI reliability as a problem of structural alignment across four interacting worlds: physical, representational, machine, and human cognitive. These worlds are connected through two interfaces: digitalization, linking physical reality to computational representations, and goal encoding, translating human cognition to the machine objectives. Together, they define the space of admissible solutions. We characterize the solution space through four attributes: existence, non-uniqueness, robustness, and interpretability and show how mismatches arise at interfaces and propagate across worlds to produce reliability failures. Applications to healthcare, energy grids, and subsurface exploration illustrate that although dominant failure modes differ across domains, for example, interpretability in healthcare, robustness in energy grids, and non-uniqueness in subsurface exploration, all originate from a shared structural mechanism. By shifting the focus from model-centric evaluation to system-level alignment, this framework offers a principled foundation for assessing and governing reliability in industrial AI systems.

Mon 28 SeptArtificial Intelligence
The gist
Artificial intelligence is being used in important areas like healthcare and energy, where mistakes can be very serious. The authors explain that judging AI only by accuracy misses problems caused by how the AI relates to the real world, the data it uses, and human intentions. They propose a framework that looks at AI reliability as the alignment of four parts: the physical world, data representations, AI models, and human goals. Problems happen when these parts don’t fit well together, and this can cause failures differently in various industries. This new way helps design and evaluate AI systems that work safely and reliably in real-world industrial settings.
Open → 2609.35400v1

Generative AI agents need resilience and care over time

Finishing the Task Is Not Enough: Evaluating Agent Resilience and Considerate Participation under Accumulating Challenge

Abstract: Sustained deployment of generative AI agents requires more than isolated task success. Agents must remain useful across repeated interactions, changing conditions, and dependencies on people within shared workflows, especially as technical, human, and operational disruptions accumulate over time. We propose operational resilience and considerate participation as two complementary aspects of evaluating such agents: the former captures how agents recover from blocked work while preserving progress and communicating their limits, and the latter captures how their adaptation accounts for affected people, role boundaries, and the surrounding workflow. Yet both remain underexplored under accumulating challenge. We study 120 simulated healthcare trajectories across two generative AI models and twelve stakeholder-derived tasks under light, medium, and heavy challenge. We compare textual action plans, prompted internal assessments, and quantitative structured workload and affect reports to examine how agent behavior and reported state change as challenge accumulates. Regarding operational resilience, agents shift from self-directed recovery toward greater human dependence, while reporting increasing workload and negative affect in structured reports but seldom expressing strain in textual responses. Regarding considerate participation, agents broaden from task-focused adaptation toward task reframing, attention to others, role-boundary adjustment, and wider coordination, with distinct patterns across actions and internal assessments. From these findings, we derive five deployment dilemmas involving persistence, attention, role boundaries, state disclosure, and escalation that require stakeholder specification, further informing technical implications for learning, situated evaluation, and embodied adaptation.

Wed 9 SeptArtificial IntelligenceHuman-Computer InteractionMultiagent Systems
The gist
Getting AI to finish a task isn’t enough for long-term use, especially in places like healthcare. The authors found that AI agents need to bounce back from problems and work well with people who depend on them. They studied how AI handles challenges over many interactions and saw that agents change how they recover and how they pay attention to others. The paper highlights important trade-offs when deploying AI to help people in ongoing, real-world workflows.
Open → 2609.10724v1