Knowledge distillation improves interpretable low-cost classifiers on tabular data

Knowledge Distillation from Neural Networks to Differentiable Logic Networks

Machine Learning

Summary

Interpretable classifiers called differentiable logic networks (DLNs) run efficiently but can be less accurate than standard neural networks. The authors show that teaching a DLN by copying knowledge from a trained neural network can boost DLN accuracy without losing its benefits like simplicity and low runtime cost. They try several ways to combine this teaching process during DLN training and test on many tabular datasets. Sometimes, the improved DLN even beats the neural network it learned from, while still being easy to understand and fast to run.

What this means in practice

  • For mobile app developers: Implement efficient and interpretable classifiers on devices with limited computational power using distilled differentiable logic networks.
  • For data scientists in finance: Deploy low-cost, interpretable models for tabular data predictions while maintaining accuracy comparable to black-box neural networks.

Authors

Hee Yun Suh, Niraj K. Jha

Abstract

Differentiable logic networks (DLNs) are interpretable, edge-friendly classifiers that learn both their logic operators and their connections through gradient descent, achieving accuracy competitive with multi-layer perceptrons at orders-of-magnitude lower inference cost. Their accuracy, however, is bounded by the problems of vanishing gradients, scalability, and lack of an ecosystem, leaving a gap relative to conventional neural networks on several datasets. We introduce knowledge distillation for DLNs to narrow this gap. We distill from a trained multi-layer perceptron teacher into a DLN student, integrating the distillation objective into a two-phase training loop without altering the architecture or the inference procedure. The teacher is discarded after training. Hence, the distilled model retains the interpretability, sparsity, and inference cost of a conventionally trained DLN. We investigate this approach through four variants: a joint hyperparameter search that optimizes the distillation weight and temperature alongside the architecture, a grid search over these two hyperparameters with the architecture fixed, and two single-phase variants that employ distillation during only the neuron-function or only the connection phase. On 20 public tabular classification tasks, distillation improves balanced-class accuracy at negligible cost to gate-level operation counts and model size, though the benefit varies by dataset. Performance improvements of up to 15% are observed, in some cases surpassing the teacher itself.