Open-weight Turkish language models tested on domain documents with limited hardware
Evaluating Open-Weight LLMs for Turkish Domain Documents Under Retrieval and Hardware Constraints
Computation and LanguageInformation RetrievalMachine Learning
Summary
This paper looks at how well different Turkish language computer programs understand long, complicated documents from specific fields like industry and public services. The researchers tested five popular models on laptops with limited graphics power, using real long reports and tricky questions. They developed a new way to tell if mistakes come from searching the right part of the document or from the model just not understanding it well enough. The results show that how well these tools work depends on the choice of model, the way documents are searched, and the computer’s memory limits.
What this means in practice
- •For software engineers: Select and configure Turkish language models that balance accuracy and hardware limits for deploying document Q&A tools locally.
- •For it infrastructure teams: Design local AI setups for Turkish domain document analysis that optimize retrieval methods and GPU memory use.
Authors
Imtiaz Ul Hassan, Öykü Akbulut, Onur Kaya, Ardhendu Behera, Swagat Kumar, Peter Matthew, Yonghuai Liu
Abstract
Most Turkish-capable large language models (LLMs) are evaluated using general-purpose benchmarks rather than long, structurally complex domain documents. This paper evaluates five open-weight 7B-8B models for Turkish document question answering under a resource-constrained local deployment setting. The primary benchmark contains 100 systematically validated questions derived from a 109-page industrial R&D report, and the evaluation protocol is replicated using a second 112-page public-sector report and an independently constructed 100-question set. All models are evaluated locally on an NVIDIA RTX 3050 laptop GPU with 6 GB VRAM using controlled prompting, decoding, and 4-bit quantisation. The principal methodological contribution is an evidence-annotated evaluation protocol that separates retrieval failure from downstream model reasoning failure without requiring additional model calls. On the primary benchmark, end-to-end accuracy ranges from 49% to 75%. Seven lexical, dense, and hybrid retrieval configurations are additionally compared using 95% Wilson intervals and exact paired McNemar tests; none significantly outperforms the character TF-IDF baseline on either document. Evidence recall saturates differently across the two reports, showing that retrieval and effective context capacity can be binding constraints for some documents but not others. These results demonstrate that model selection, retrieval behaviour, and hardware limits must be evaluated separately when deploying open-weight LLMs for Turkish domain documents.