DKnown-Jev improves accuracy in judging rule-based knowledge answers
DKnown-Jev: Fine-Tuning and Evaluation of a JEV-Style Model for Tasks Involving Normative Knowledge
Artificial Intelligence
Summary
Some tasks need checking if paragraphs properly answer questions about rules or official facts. The paper’s authors created DKnown-Jev, a model that decides if a paragraph is a good answer or not, using different sizes of language models fine-tuned on verified data. They tested several versions and found that their largest model strongly outperformed an earlier system called Jev. This helps in reliably finding correct and relevant answers in fields that use official rules or procedures.
What this means in practice
- •For legal technology teams: Automate identification of paragraphs that properly address eligibility and procedural questions in law documents using tuned Qwen-based models.
- •For policy compliance analysts: Improve automated scoring of document sections to verify adherence to published rules and requirements in regulatory text analysis.
Authors
Qi Li, Jianjun Xu, Pingtao Wei
Abstract
Tasks involving normative knowledge include concept interpretation, eligibility determination, procedure explanation, material requirements, and service-outcome assessment, all of which require rules and facts published, maintained, or recorded by authorized entities. Whether a candidate paragraph can answer a question therefore depends on applicability and answer completeness as well as topical relevance. We develop DKnown-Jev to classify the normative usability of candidate paragraphs into two classes. The 2B and 27B configurations are fine-tuned directly from the native Qwen3.5-2B and Qwen3.8-27B base models, respectively. The models combine question conditions, article metadata, and paragraph content, adopt the classes Answer Evidence (AE) and Non-answer Evidence (NE), and undergo domain adaptation using human-verified training data. Five configurations are evaluated on the same test set of 1,120 human-verified examples. DKnown-Jev 27B trained for one epoch achieves the highest accuracy of 98.75% and macro-F1 of 98.60%, exceeding Jev by 6.07 and 6.76 percentage points, respectively. DKnown-Jev 2B trained for one epoch achieves 92.86% accuracy and 92.18% macro-F1, the best aggregate results among the tested 2B configurations. Its gains over Jev are 0.18 and 0.33 percentage points on this test set, corresponding to only two additional correct predictions, which is insufficient to establish a stable advantage. The 27B one-epoch, 27B two-epoch, and 2B one-epoch configurations all exceed Jev on both metrics on this test set. Class-level analysis shows that the 27B two-epoch configuration has the highest AE recall, while the 27B one-epoch configuration has the highest NE recall. This study provides practical and empirical evidence for developing and applying candidate-scoring models to tasks involving normative knowledge.