Tree based rules improve neural network training performance

TreePIN$^{2}$: Tree-Prior Initialized Neural Networks

Machine LearningArtificial Intelligence

Summary

Neural networks usually start with random settings and learn from data over time. This paper presents a way to begin neural network training by using the rules from decision trees — a simpler type of model — to set the network’s initial values. The authors translate the tree’s decisions into network parameters, so the network starts with meaningful knowledge. Their tests show this approach helps the network perform better early on and often reaches stronger accuracy after training.

What this means in practice

  • For machine learning engineers: Initialize neural networks using decision tree insights to improve early training speed and accuracy across varied tasks.
  • For financial data analysts: Use tree-based neural network starters to get more accurate regression predictions in financial modeling tasks.

Authors

Yanrong Wang, Qiming Sun, Haoyang Chen

Abstract

Fitted decision trees provide predictive knowledge for neural network initialization. We present Tree-Prior Initialized Neural Networks (TreePIN$^{2}$), a method that translates split rules, decision paths and leaf predictions into initial network weights and biases for classification and regression. Sigmoid units encode split responses, hidden units combine them along decision paths, and leaf predictions initialize the output layer. We calibrate split softness using local training margins, sample support and impurity reduction, relating the initial responses to the data at each node. All weights and biases remain trainable, allowing gradient-based optimization to refine the transferred decisions and their combinations. Across six classification and six regression tasks with twenty paired seeds, TreePIN$^{2}$ improves mean initial and early validation performance over initialization baselines on most tasks. After fifty epochs, it achieves the best or tied-best mean validation performance among the six compared methods on ten tasks. Paired ablations show how split calibration and leaf-informed output initialization contribute to the starting predictor and its subsequent learning.