Recurrent-depth language models improve clinical text tasks
Exploring Recurrent-Depth Language Models for Multitask Clinical Text Processing
Computation and Language
Summary
Clinical text processing involves tasks like extracting information, classifying data, and answering questions about medical records. The authors tested smaller language models that use a type of repeating computation called recurrent-depth to see if one model could handle many tasks efficiently. They found these models often performed better or competitively compared to standard transformer models, especially for reasoning and question answering. Their work suggests studying how these repeating computations help with understanding medical texts and sharing knowledge across tasks.
What this means in practice
- •For hospital data teams: Deploy unified language models that handle multiple clinical text processing tasks, improving efficiency and consistency in medical data workflows.
- •For legal document analysts: Use recurrent-depth language models to perform complex reasoning and question answering on legal texts involving dense, specialized language.
Authors
Xi Yang, Guodong Liu, Cheng Peng, Kaleb Smith, Yingying Zhang, Xing He, Yi Guo, Jiang Bian, Haoteng Tang, Yonghui Wu
Abstract
Clinical natural language processing includes diverse tasks such as information extraction, classification, and question answering. Developing one model for all tasks could simplify deployment and improve knowledge transfer. We evaluated whether small recurrent-depth language models can meet this need through joint fine-tuning and compared their performance with standard transformers. We fine-tuned Ouro1.4B-Thinking, Ouro-2.6B-Thinking, HRM-Text-1B, Qwen3-1.7B, and Qwen3-8B on 9 datasets covering information extraction, classification, inference, and question answering, and evaluated them on four additional datasets excluded from training. Across these 13 datasets, Ouro-1.4B-Thinking achieved higher performances than Qwen3-1.7B on 11 datasets; Ouro-2.6BThinking was competitive with Qwen3-8B on most tasks. Ouro models showed the largest gains on inference and question answering, whereas HRM-Text-1B showed mixed results. These findings support further study of recurrent-depth models for clinical tasks through controlled experiments to isolate architectural effects and investigate how recurrent computation influences reasoning, knowledge acquisition, and generalization.