Jiancheng Dong

Threshold Filtering Packing for Supervised Fine-Tuning: Training Related Samples within Packs

Packing for Supervised Fine-Tuning (SFT) in autoregressive models involves concatenating data points of varying lengths until reaching the designed maximum length to facilitate GPU processing. However, randomly concatenating data points can lead to cross-contamination of sequences due to the signifi…