Split Group Knockoffs: Controlling False Discovery Rate in Transformational Group Sparsity

Abstract

Controlling the false discovery rate (FDR) under complex sparsity structures remains a fundamental challenge in large language model (LLM) analysis. Motivated by multiple comparison problems in LLMs, we consider a setting in which sparsity arises at the group level after a linear transformation of model parameters. We propose Split Group Knockoffs (SGKs), a general framework for group-wise variable selection under grouped transformational sparsity that extends the Split Knockoff procedure to grouped transformed variables. We establish theoretical guarantees for group-level FDR control and support recovery consistency, addressing challenges induced by group-wise penalties in transformed spaces. Applying SGK to LLM behavior auditing experiment reveals that model disagreement is not uniform across subjects, but instead concentrates in domains with greater semantic and reasoning complexity, where SGK effectively distinguishes genuine behavioral deviations from surface-level performance variation.