Cacheable decision model struggles to follow rules accurately
Can a Cacheable Decision Model Follow Rules?
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
This paper looks at how a particular type of decision model, called Certo, scores possible actions based on text and rules. The authors compare two ways of designing it: one that processes all information together (accurate but costly), and another that encodes candidates independently so results can be reused (cheaper but less precise). They found that the cheaper version initially lost almost all ability to follow rules, but with careful training it regained some skill on synthetic tests. However, on real-world rules, the cheaper model still performed worse than the full joint approach, so the trade-off between cost and accuracy remains.
What this means in practice
- •For software engineers: Build decision systems that cache candidate data to reduce computation, with careful training to maintain rule sensitivity on synthetic tasks.
- •For legal tech developers: Improve rule-compliance checking tools by using joint scoring to accurately handle complex legal rules, accepting higher computation costs.
Authors
Dushyant Rajput, Nirdesh Chauhan, Siddharth Kosaraju
Abstract
Certo is a small non-generative decision model (Qwen3-4B): it scores candidate actions from their text and returns a probability, instead of generating an answer. The accurate design reads the state, the rules, and each candidate together (a joint scorer), so cost grows with the menu. Independent encoding lets each candidate be encoded once and reused across states (about 5x cheaper at 77 candidates), but separates state from candidate. We ask how much rule-sensitivity survives that move, and whether it can be trained back. Four experiments on Certo: (1) the tested conversion to cacheable scoring loses rule-sensitivity (recall@1 1.00 -> 0.24) while the joint scorer holds 1.00, and a shortlist+rerank rescue fails; (2) targeted counterfactual supervision restores strong performance on held-out synthetic rule tasks (paraphrase, counterfactual, composition; reproducible across seeds), though we do not isolate whether predictions depend on the supplied rule; (3) on real rules the added benefit is not established -- after fixing a truncation confound, the joint scorer wins significantly on the short tier (0.861 vs 0.500) and directionally on the hard tier (0.655 vs 0.483, n=29); (4) a matched cross-domain real-prose mixture did not help and reduced contract accuracy (-9.3, -16.2 points). A cacheable encoder can be made rule-sensitive on its training distribution, but transfer to unseen-source real rules is not established; the joint scorer keeps an edge at the cost of caching.