AIConfPaper
← Search

Feiyang Kang

10 accepted papers

2026

How Text Quality Interventions Reshape Neural Scaling Laws for LLMs: Empirical Study

Newsha Ardalani, Feiyang Kang, Michael Kuchnik, Mostafa Elhoushi, Shubhabrata Sengupta, Shang-Wen Li +1
ICLR 2026poster

Neural scaling laws are widely used for performance projection and resource planning, yet their sensitivity to data quality interventions remains poorly understood. We present an empirical study of how interventions—deduplication, heuristic filtering, and LLM-guided rewriting—reshape scaling behavi…

Cited by 0Source ↗Scholar ↗
2026

Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead

Feiyang Kang, Michael Kuchnik, Karthik Padthe, Marin Vlastelica, Ruoxi Jia, Carole-Jean Wu +1
ICLR 2026poster

In post-training for reasoning Large Language Models (LLMs), the current state of practice trains LLMs in two independent stages: Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR, shortened as "RL" below). In this work, we challenge whether high SFT scores transl…

Cited by 0Source ↗Scholar ↗
2025

A Sustainable AI Economy Needs Data Deals That Work for Generators

Ruoxi Jia, Luis Oala, Wenjie Xiong, Suqin Ge, Jiachen T. Wang, Feiyang Kang +1
NeurIPS 2025poster

We argue that the machine learning value chain is structurally unsustainable due to an economic data processing inequality: each state in the data cycle from inputs to model weights to synthetic outputs refines technical signal but strips economic equity from data generators. We show, by analyzing s…

Cited by 0Source ↗Scholar ↗
2025

Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls

Feiyang Kang, Newsha Ardalani, Michael Kuchnik, Youssef Emad, Mostafa Elhoushi, Shubhabrata Sengupta +4
EMNLP 2025

Training data plays a crucial role in Large Language Models (LLM) scaling, yet high quality data is of limited supply. Synthetic data techniques offer a potential path toward sidestepping these limitations.We conduct a large-scale empirical investigation (>1000 LLMs with >100k GPU hours) using a uni

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

FASTTRACK: Reliable Fact Tracing via Clustering and LLM-Powered Evidence Validation

Si Chen, Feiyang Kang, Ning Yu, Ruoxi Jia
EMNLP 2024finding

Fact tracing seeks to identify specific training examples that serve as the knowledge source for a given query. Existing approaches to fact tracing rely on assessing the similarity between each training sample and the query along a certain dimension, such as lexical similarity, gradient, or embeddin…

Cited by 0Source ↗PDF ↗code ↗Scholar ↗
2024

Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs

Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du +2
ICLR 2024poster

This work focuses on leveraging and selecting from vast, unlabeled, open data to *pre-fine-tune* a pre-trained language model. The goal is to minimize the need for costly domain-specific data for subsequent fine-tuning while achieving desired performance levels. While many data selection algorithms…

Cited by 15Source ↗PDF ↗Scholar ↗
2024

The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes

Myeongseob Ko, Feiyang Kang, Weiyan Shi, Ming Jin, Zhou Yu, Ruoxi Jia
CVPR 2024poster

Large-scale black-box models have become ubiquitous across numerous applications. Understanding the influence of individual training data sources on predictions made by these models is crucial for improving their trustworthiness. Current influence estimation techniques involve computing gradients fo…

Cited by 4Source ↗PDF ↗code ↗Scholar ↗
2023

LAVA: Data Valuation without Pre-Specified Learning Algorithms

Hoang Anh Just, Feiyang Kang, Tianhao Wang, Yi Zeng, Myeongseob Ko, Ming Jin +1
ICLR 2023top-25%

Traditionally, data valuation is posed as a problem of equitably splitting the validation performance of a learning algorithm among the training data. As a result, the calculated data values depend on many design choices of the underlying learning algorithm. However, this dependence is undesirable f…

Cited by 69Source ↗PDF ↗code ↗Scholar ↗
2023

Performance Scaling via Optimal Transport: Enabling Data Selection from Partially Revealed Sources

Feiyang Kang, Hoang Anh Just, Anit Kumar Sahu, Ruoxi Jia
NeurIPS 2023poster

Traditionally, data selection has been studied in settings where all samples from prospective sources are fully revealed to a machine learning developer. However, in practical data exchange scenarios, data providers often reveal only a limited subset of samples before an acquisition decision is made…

Cited by 14Source ↗PDF ↗Scholar ↗
2023

Towards Robustness Certification Against Universal Perturbations

Yi Zeng, Zhouxing Shi, Ming Jin, Feiyang Kang, Lingjuan Lyu, Cho-Jui Hsieh +1
ICLR 2023poster

In this paper, we investigate the problem of certifying neural network robustness against universal perturbations (UPs), which have been widely used in universal adversarial attacks and backdoor attacks. Existing robustness certification methods aim to provide robustness guarantees for each sample w…

Cited by 13Source ↗PDF ↗Scholar ↗