ChunkRank improves text chunking and answer picking for language models
ChunkRank: Model-Aware Text Chunking and Abstention-Aware Answer Selection for LLM Pipelines
Computation and Language
Summary
When working with big language models, it’s important to split up long texts into chunks that fit the model’s limits. The authors created ChunkRank, a tool that uses the specific way a model breaks up text to make chunks just right and avoids wasting space. They found that simply picking the first valid answer from those chunks works better than trying to rank answers by content. ChunkRank helps especially when texts are longer than the model’s context window, making it easier to handle big documents in parts.
What this means in practice
- •For software engineers: Implement chunking that fits the exact token limits of various language models to avoid wasted context space or errors from overflow.
- •For technical writers: Improve automated document summarization workflows by segmenting texts according to model tokenization and selecting non-empty answers efficiently.
Authors
Amit Nautiyal, Ayush Bhatt, Gaurav Nautiyal
Abstract
We present ChunkRank, an open-source Python library that derives chunk boundaries from a target model's tokenizer and context window, and selects an answer among candidates produced independently per chunk. It ships a validated registry of 90 models across 15 providers and six answer-selection methods, and needs only three core dependencies. For chunking, ChunkRank avoids context-window overflow automatically from the model name, whereas character-based splitters overflow or waste the budget, and a fidelity study across 11 languages shows why token-exact budgets matter beyond English. For answer selection we report a negative result: on NaturalQuestions, TriviaQA and HotpotQA, with extractive and generative readers, no content-based ranker reliably beats taking the first non-empty answer. The reason is reader abstention on chunks that lack the answer, not answer position. A long-context baseline shows that chunking matches single-call reading on single-hop questions, so ChunkRank targets small-window and beyond-window settings. Code, registry and evaluation harness are released.