Boruta gene selection improves with uncapped tree depth and quality filtering
Tree Depth and Sample-Quality-Associated Features: Two Hidden Factors in Boruta Selection for RNA-Seq Data Classification
Machine Learning
Summary
Selecting important genes from RNA sequencing data helps understand diseases like cancer. The authors found that limiting the depth of decision trees in the popular Boruta method changes which genes are picked and lowers accuracy. They showed that allowing deeper trees and filtering genes related to sample quality improves the gene panel selection without losing accuracy. This helps build better tools for classifying cancer types based on gene expression.
What this means in practice
- •For cancer diagnostic developers: Use uncapped tree depth with Boruta and pre-filter genes by sample quality to improve gene panels for cancer subtype classification from RNA-Seq data.
- •For biomedical data engineers: Optimize RNA-Seq gene selection pipelines by adjusting Random Forest tree depth dynamically and filtering out quality-associated genes within classes to maintain accuracy.
Authors
Tareque Mohmud Chowdhury, Abu Raihan Mostofa Kamal
Abstract
Boruta is a widely used method for selecting gene panels from RNA sequencing data, and its most used implementation recommends Random Forest trees of depth 3 to 7, an untested setting. We tested whether this setting changes which genes are selected, using pan-cancer data from The Cancer Genome Atlas (10,469 tumors, 33 cancer types) and breast cancer PAM50 subtyping (981 tumors, 5 subtypes). A leakage-free protocol fitted every label-dependent step inside training folds, measured every effect against seed-to-seed variation, and held out a locked test set per task. In pan-cancer data, shallow trees changed the selection far more than a new seed did: the top-500 overlap with unlimited-depth selections was 0.37 at depth 5, against 0.84 to 0.93 between seeds at depth 5 or unlimited depth. Shallow trees confirmed 3.5 times fewer genes, missed markers of large classes and produced less accurate 50-gene panels (balanced accuracy 0.809 vs 0.889). The effect vanished once the cap reached about 60 to 75% of the depth that unrestricted trees grow to, set mainly by the number of classes, not sample size, and leaving depth uncapped cost no accuracy in either task. The effect replicated in a second cross-validation repeat and under permutation importance. About a third of pre-filtered candidates also tracked sample quality within class; a within-class filter removed them without loss of test accuracy (balanced accuracy 0.943 vs 0.942 with 1,000 genes). We recommend uncapped trees, or a cap chosen from the realized tree depth, and a within-class quality filter before Boruta.