TabKD: Tabular Knowledge Distillation Through Interaction Diversity of Learned Feature Bins

Abstract

Data-free knowledge distillation enables model compression without original training data, criti- cal for privacy-sensitive tabular domains. How- ever, existing methods fail on tabular data because they ignore feature interactions, the fundamen- tal way tabular models encode predictive knowl- edge. We identify interaction diversity, system- atic coverage of feature combinations, as the es- sential requirement for effective tabular distilla- tion. To operationalize this insight, we propose TabKD, which learns adaptive feature bins aligned with teacher decision boundaries, then generates synthetic queries that maximize pairwise interac- tion coverage. Across 4 benchmark datasets and 4 teacher architectures, TabKD achieves highest student-teacher agreement in 14 out of 16 configu- rations, outperforming 5 state-of-the-art baselines. We further show that interaction coverage strongly correlates with distillation quality, validating our core hypothesis. Our work establishes interaction- focused exploration as a principled framework for tabular model extraction.