← Search

Changyou Chen

73 accepted papers

2026

ALPHA: Action-Based Learning for Pluralistic Human Alignment in Large Language Models

AAAI 2026technical

Large language models are widely used, but aligning them with societal values remains challenging. Current approaches often rely on human annotations, which are hard to scale, or synthetic data produced by models that may themselves be misaligned, making it difficult to capture genuine public opinio

Cited by 0SourcePDFScholar
2026

KGOT: Unified Knowledge Graph and Optimal Transport Pseudo-Labeling for Molecule-Protein Interaction Prediction

ICLR 2026poster

Predicting molecule-protein interactions (MPIs) is a fundamental task in computational biology, with crucial applications in drug discovery and molecular function annotation. However, existing MPI models face two major challenges. First, the scarcity of labeled molecule-protein pairs significantly l…

Cited by 0SourceScholar
2026

SFT Doesn’t Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs

ICLR 2026poster

Supervised Fine-Tuning (SFT) on domain-specific datasets is a common approach to adapt Large Language Models (LLMs) to specialized tasks but is often believed to degrade their general capabilities. In this work, we revisit this trade-off and present both empirical and theoretical insights. First, we…

Cited by 0SourceScholar
2026

Social Agents: Collective Intelligence Improves LLM Predictions

ICLR 2026poster

In human society, collective decision making has often outperformed the judgment of individuals. Classic examples range from estimating livestock weights to predicting elections and financial markets, where averaging many independent guesses often yields results more accurate than experts. These suc…

Cited by 0SourceScholar
2026

Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

CVPR 2026

Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is challenging, as some samples may be noisy while others are subject to teacher uncerta

Cited by 0SourcecodeScholar
2025

A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation

COLING 2025main

Large multimodal models still struggle with text-rich images because of inadequate training data. Self-Instruct provides an annotation-free way for generating instruction data, but its quality is poor, as multimodal alignment remains a hurdle even for the largest models. In this work, we propose LLa…

2025

CLAP-S: Support Set Based Adaptation for Downstream Fiber-optic Acoustic Recognition

ICASSP 2025accepted

Contrastive Language-Audio Pretraining (CLAP) models have demonstrated unprecedented performance in various acoustic signal recognition tasks. Fiber-optic-based acoustic recognition is one of the most important downstream tasks and plays a significant role in environmental sensing. Adapting CLAP for…

Cited by 0SourceScholar
2025

Measuring And Improving Engagement of Text-to-Image Generation Models

ICLR 2025poster

Recent advances in text-to-image generation have achieved impressive aesthetic quality, making these models usable for both personal and commercial purposes. However, in the fields of marketing and advertising, images are often created to be more engaging, as reflected in user behaviors such as incr…

2025

Multi-Modal Multi-Task Unified Embedding Model (M3T-UEM): A Task-Adaptive Representation Learning Framework

ICCV 2025poster

We present Multi-Modal Multi-Task Unified Embedding Model (M3T-UEM), a framework that advances vision-language matching and retrieval by leveraging a large language model (LLM) backbone. While concurrent LLM-based approaches like VLM2VEC, MM-Embed, NV-Embed, and MM-GEM have demonstrated impressive c…

2025

Multimodal LLMs as Customized Reward Models for Text-to-Image Generation

ICCV 2025poster

We introduce LLaVA-Reward, an efficient reward model designed to automatically evaluate text-to-image (T2I) generations across multiple perspectives, leveraging pretrained multimodal large language models (MLLMs). Existing MLLM-based approaches require instruction-following data for supervised fine-…

2025

SPRO: Improving Image Generation via Self-Play

NeurIPS 2025poster

Recent advances in diffusion models have dramatically improved image fidelity and diversity. However, aligning these models with nuanced human preferences -such as aesthetics, engagement, and subjective appeal remains a key challenge due to the scarcity of large-scale human annotations. Collecting s…

Cited by 0SourceScholar
2025

SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding

ICLR 2025poster

Multimodal large language models (MLLMs) have recently shown great progress in text-rich image understanding, yet they still struggle with complex, multi-page visually-rich documents. Traditional methods using document parsers for retrieval-augmented generation suffer from performance and efficiency…

Cited by 0SourcePDFScholar
2025

Teaching Human Behavior Improves Content Understanding Abilities Of VLMs

ICLR 2025poster

Communication is defined as "*Who* says *what* to *whom* with *what* effect." A message from a communicator generates downstream receiver effects, also known as behavior. Receiver behavior, being a downstream effect of the message, carries rich signals about it. Even after carrying signals about the…

2024

A probability contrastive learning framework for 3D molecular representation learning

NeurIPS 2024poster

Contrastive Learning (CL) plays a crucial role in molecular representation learning, enabling unsupervised learning from large scale unlabeled molecule datasets. It has inspired various applications in molecular property prediction and drug design. However, existing molecular representation learning…

Cited by 0SourcePDFScholar
2024

AUC-CL: A Batchsize-Robust Framework for Self-Supervised Contrastive Representation Learning

ICLR 2024poster

Self-supervised learning through contrastive representations is an emergent and promising avenue, aiming at alleviating the availability of labeled data. Recent research in the field also demonstrates its viability for several downstream tasks, henceforth leading to works that implement the contrast…

Cited by 3SourcePDFScholar
2024

Diffusion Models for Multi-Task Generative Modeling

ICLR 2024poster

Diffusion-based generative modeling has been achieving state-of-the-art results on various generation tasks. Most diffusion models, however, are limited to a single-generation modeling. Can we generalize diffusion models with the ability of multi-modal generative training for more generalizable mode…

Cited by 7SourcePDFScholar
2024

Large Content And Behavior Models To Understand, Simulate, And Optimize Content And Behavior

ICLR 2024spotlight

Shannon and Weaver's seminal information theory divides communication into three levels: technical, semantic, and effectiveness. While the technical level deals with the accurate reconstruction of transmitted symbols, the semantic and effectiveness levels deal with the inferred meaning and its effec…

2024

TRINS: Towards Multimodal Language Models that Can Read

CVPR 2024poster

Large multimodal language models have shown remarkable proficiency in understanding and editing images. However a majority of these visually-tuned models struggle to comprehend the textual content embedded in images primarily due to the limitation of training data. In this work we introduce TRINS: a…

Cited by 9SourcePDFScholar
2024

TextLap: Customizing Language Models for Text-to-Layout Planning

EMNLP 2024finding

Automatic generation of graphical layouts is crucial for many real-world applications, including designing posters, flyers, advertisements, and graphical user interfaces. Given the incredible ability of Large language models (LLMs) in both natural language understanding and generation, we believe th…

2024

Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints

ICLR 2024poster

Controllable layout generation refers to the process of creating a plausible visual arrangement of elements within a graphic design (*e.g.*, document and web designs) with constraints representing design intentions. Although recent diffusion-based models have achieved state-of-the-art FID scores, th…

2023

A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them In Zero Shot

EMNLP 2023long main

Multimedia content, such as advertisements and story videos, exhibit a rich blend of creativity and multiple modalities. They incorporate elements like text, visuals, audio, and storytelling techniques, employing devices like emotions, symbolism, and slogans to convey meaning. There is a dearth of l…

Cited by 0SourceScholar
2023

AUC Maximization for Low-Resource Named Entity Recognition

AAAI 2023technical

Current work in named entity recognition (NER) uses either cross entropy (CE) or conditional random fields (CRF) as the objective/loss functions to optimize the underlying NER model. Both of these traditional objective functions for the NER problem generally produce adequate performance when the dat…

2023

Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reduction

ICML 2023poster

Due to the often limited communication bandwidth of edge devices, most existing federated learning (FL) methods randomly select only a subset of devices to participate in training at each communication round. Compared with engaging all the available clients, such a random-selection mechanism could l…

Cited by 58SourcePDFScholar
2023

Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels

NeurIPS 2023poster

Learning from noisy labels is an important and long-standing problem in machine learning for real applications. One of the main research lines focuses on learning a label corrector to purify potential noisy labels. However, these methods typically rely on strict assumptions and are limited to certai…

2023

Learning Unnormalized Statistical Models via Compositional Optimization

ICML 2023poster

Learning unnormalized statistical models (e.g., energy-based models) is computationally challenging due to the complexity of handling the partition function. To eschew this complexity, noise-contrastive estimation (NCE) has been proposed by formulating the objective as the logistic loss of the real…

Cited by 4SourcePDFScholar
2023

Persuasion Strategies in Advertisements

AAAI 2023technical

Modeling what makes an advertisement persuasive, i.e., eliciting the desired response from consumer, is critical to the study of propaganda, social psychology, and marketing. Despite its importance, computational modeling of persuasion in computer vision is still in its infancy, primarily due to the…

2023

ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models

ACL 2023short

Knowledge Distillation (KD) is one of the most effective approaches to deploying large-scale pre-trained language models in low-latency environments by transferring the knowledge contained in the large-scale models to smaller student models. Prior KD approaches use the soft labels and intermediate a…

Cited by 23SourcePDFScholar
2023

Shifted Diffusion for Text-to-Image Generation

CVPR 2023poster

We present Corgi, a novel method for text-to-image generation. Corgi is based on our proposed shifted diffusion model, which achieves better image embedding generation from input text. Different from the baseline diffusion model used in DALL-E 2, our method seamlessly encodes prior knowledge of the…

2023

Understanding and Constructing Latent Modality Structures in Multi-Modal Representation Learning

CVPR 2023poster

Contrastive loss has been increasingly used in learning representations from multiple modalities. In the limit, the nature of the contrastive loss encourages modalities to exactly match each other in the latent space. Yet it remains an open question how the modality alignment affects the downstream…

Cited by 54SourcePDFScholar
2022

Hardness-guided domain adaptation to recognise biomedical named entities under low-resource scenarios

EMNLP 2022main

Domain adaptation is an effective solution to data scarcity in low-resource scenarios. However, when applied to token-level tasks such as bioNER, domain adaptation methods often suffer from the challenging linguistic characteristics that clinical narratives possess, which leads to unsatsifactory per…

2022

MINIMAL: Mining Models for Universal Adversarial Triggers

AAAI 2022technical

It is well known that natural language models are vulnerable to adversarial attacks, which are mostly input-specific in nature. Recently, it has been shown that there also exist input-agnostic attacks in NLP models, called universal adversarial triggers. However, existing methods to craft universal…

Cited by 5SourcePDFScholar
2022

TiGAN: Text-Based Interactive Image Generation and Manipulation

AAAI 2022technical

Using natural-language feedback to guide image generation and manipulation can greatly lower the required efforts and skills. This topic has received increased attention in recent years through refinement of Generative Adversarial Networks (GANs); however, most existing works are limited to single-r…

Cited by 20SourcePDFScholar
2022

Towards Language-Free Training for Text-to-Image Generation

CVPR 2022poster

One of the major challenges in training text-to-image generation models is the need of a large number of high-quality text-image pairs. While image samples are often easily accessible, the associated text description typically requires careful human captioning, which is particularly time- and cost-c…

Cited by 307PDFScholar
2022

Why do We Need Large Batchsizes in Contrastive Learning? A Gradient-Bias Perspective

NeurIPS 2022accept

Contrastive learning (CL) has been the de facto technique for self-supervised representation learning (SSL), with impressive empirical success such as multi-modal representation learning. However, traditional CL loss only considers negative samples from a minibatch, which could cause biased gradient…

Cited by 40SourcePDFScholar
2021

Integrating Semantics and Neighborhood Information with Graph-Driven Generative Models for Document Retrieval

ACL 2021long

With the need of fast retrieval speed and small memory footprint, document hashing has been playing a crucial role in large-scale information retrieval. To generate high-quality hashing code, both semantics and neighborhood information are crucial. However, most existing methods leverage only one of…

2021

MixKD: Towards Efficient Distillation of Large-scale Language Models

ICLR 2021poster

Large-scale language models have recently demonstrated impressive empirical performance. Nevertheless, the improved results are attained at the price of bigger models, more power consumption, and slower inference, which hinder their applicability to low-resource (both memory and computation) platfor…

Cited by 90SourcePDFScholar
2021

Unsupervised Hashing with Contrastive Information Bottleneck

IJCAI 2021poster

Many unsupervised hashing methods are implicitly established on the idea of reconstructing the input data, which basically encourages the hashing codes to retain as much information of original data as possible. However, this requirement may force the models spending lots of their effort on reconstr…

2020

Bayesian Meta Sampling for Fast Uncertainty Adaptation

ICLR 2020poster

Meta learning has been making impressive progress for fast model adaptation. However, limited work has been done on learning fast uncertainty adaption for Bayesian modeling. In this paper, we propose to achieve the goal by placing meta learning on the space of probability measures, inducing the conc…

Cited by 25SourcecodeScholar
2020

Bayesian Multi-type Mean Field Multi-agent Imitation Learning

NeurIPS 2020spotlight

Multi-agent Imitation learning (MAIL) refers to the problem that agents learn to perform a task interactively in a multi-agent system through observing and mimicking expert demonstrations, without any knowledge of a reward function from the environment. MAIL has received a lot of attention due to pr…

Cited by 20SourcePDFScholar
2020

Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

ICLR 2020talk

The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cycli…

Cited by 357SourcecodeScholar
2020

Feature Quantization Improves GAN Training

ICML 2020poster

The instability in GANs’ training has been a long-standing problem despite remarkable research efforts. We identify that instability issues stem from difficulties of performing feature matching with mini-batch statistics, due to a fragile balance between the fixed target distribution and the progres…

2020

Nested-Wasserstein Self-Imitation Learning for Sequence Generation

AISTATS 2020poster

Reinforcement learning (RL) has been widely studied for improving sequence-generation models. However, the conventional rewards used for RL training typically cannot capture sufficient semantic information and therefore render model bias. Further, the sparse and delayed rewards make RL exploration i…

Cited by 8SourcePDFScholar
2020

Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory

AISTATS 2020poster

Particle-optimization-based sampling (POS) is a recently developed effective sampling technique that interactively updates a set of particles. A representative algorithm is the Stein variational gradient descent (SVGD). We prove, under certain conditions, SVGD experiences a theoretical pitfall, {\i…

Cited by 51SourcePDFScholar
2019

Adversarial Learning of a Sampler Based on an Unnormalized Distribution

AISTATS 2019poster

Fundamental aspects of adversarial learning are investigated, with learning based on samples from the target distribution (conventional GAN setup). With insights so garnered, adversarial learning is extended to the case for which one has access to an unnormalized form $u(x)$ of the target density fu…

2019

Certified Adversarial Robustness with Additive Noise

NeurIPS 2019poster

The existence of adversarial data examples has drawn significant attention in the deep-learning community; such data are seemingly minimally perturbed relative to the original data, but lead to very different outputs from a deep-learning algorithm. Although a significant body of work on developing d…

2019

Differentially Private Empirical Risk Minimization with Non-convex Loss Functions

ICML 2019oral

We study the problem of Empirical Risk Minimization (ERM) with (smooth) non-convex loss functions under the differential-privacy (DP) model. Existing approaches for this problem mainly adopt gradient norms to measure the error, which in general cannot guarantee the quality of the solution. To addres…

Cited by 105SourcePDFScholar
2019

Improving Sequence-to-Sequence Learning via Optimal Transport

ICLR 2019poster

Sequence-to-sequence models are commonly trained via maximum likelihood estimation (MLE). However, standard MLE training considers a word-level objective, predicting the next word given the previous ground-truth partial sentence. This procedure focuses on modeling local syntactic patterns, and may f…

Cited by 110SourcePDFScholar
2019

On Connecting Stochastic Gradient MCMC and Differential Privacy

AISTATS 2019poster

Concerns related to data security and confidentiality have been raised when applying machine learning to real-world applications. Differential privacy provides a principled and rigorous privacy guarantee for machine learning models. While it is common to inject noise to design a model satisfying a r…

Cited by 51SourcePDFScholar
2019

Scalable Thompson Sampling via Optimal Transport

AISTATS 2019poster

Thompson sampling (TS) is a class of algorithms for sequential decision-making, which requires maintaining a posterior distribution over a reward model. However, calculating exact posterior distributions is intractable for all but the simplest models. Consequently, how to computationally-efficiently…

2019

Text-Based Interactive Recommendation via Constraint-Augmented Reinforcement Learning

NeurIPS 2019accepted

Text-based interactive recommendation provides richer user preferences and has demonstrated advantages over traditional interactive recommender systems. However, recommendations can easily violate preferences of users from their past natural-language feedback, since the recommender needs to explore…

2018

Continuous-Time Flows for Efficient Inference and Density Estimation

ICML 2018oral

Two fundamental problems in unsupervised learning are efficient inference for latent-variable models and robust density estimation based on large amounts of unlabeled data. Algorithms for the two tasks, such as normalizing flows and generative adversarial networks (GANs), are often developed indepen…

2018

Learning Structural Weight Uncertainty for Sequential Decision-Making

AISTATS 2018poster

Learning probability distributions on the weights of neural networks (NNs) has recently proven beneficial in many applications. Bayesian methods, such as Stein variational gradient descent (SVGD), offer an elegant framework to reason about NN model uncertainty. However, by assuming independent Gauss…

2018

Symmetric Variational Autoencoder and Connections to Adversarial Learning

AISTATS 2018poster

A new form of the variational autoencoder (VAE) is proposed, based on the symmetric Kullback- Leibler divergence. It is demonstrated that learn- ing of the resulting symmetric VAE (sVAE) has close connections to previously developed adversarial-learning methods. This relationship helps unify the pre…

Cited by 0SourcePDFScholar
2017

ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching

NeurIPS 2017poster

We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching. Within a framework of conditional entropy, we propose both adversarial and non-adversarial approaches to learn desirable matched joint distributions for unsupervised and…

2017

Learning Structured Weight Uncertainty in Bayesian Neural Networks

AISTATS 2017poster

Deep neural networks (DNNs) are increasingly popular in modern machine learning. Bayesian learning affords the opportunity to quantify posterior uncertainty on DNN model parameters. Most existing work adopts independent Gaussian priors on the model weights, ignoring possible structural information.…

Cited by 153SourcePDFScholar
2016

Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization

AISTATS 2016poster

Stochastic gradient Markov chain Monte Carlo (SG-MCMC) methods are Bayesian analogs to popular stochastic optimization methods; however, this connection is not well studied. We explore this relationship by applying simulated annealing to an SG-MCMC algorithm. Furthermore, we extend recent SG-MCMC me…

2016

Learning Weight Uncertainty With Stochastic Gradient MCMC for Shape Classification

CVPR 2016spotlight

Learning the representation of shape cues in 2D & 3D objects for recognition is a fundamental task in computer vision. Deep neural networks (DNNs) have shown promising performance on this task. Due to the large variability of shapes, accurate recognition relies on good estimates of model uncertain…

Cited by 62PDFScholar
2016

Nonlinear Statistical Learning with Truncated Gaussian Graphical Models

ICML 2016poster

We introduce the truncated Gaussian graphical model (TGGM) as a novel framework for designing statistical models for nonlinear learning. A TGGM is a Gaussian graphical model (GGM) with a subset of variables truncated to be nonnegative. The truncated variables are assumed latent and integrated out to…

Cited by 19SourcePDFScholar
2016

Stochastic Gradient MCMC with Stale Gradients

NeurIPS 2016poster

Stochastic gradient MCMC (SG-MCMC) has played an important role in large-scale Bayesian learning, with well-developed theoretical convergence properties. In such applications of SG-MCMC, it is becoming increasingly popular to employ distributed systems, where stochastic gradients are computed based…

Cited by 34SourcePDFScholar
2016

Towards Unifying Hamiltonian Monte Carlo and Slice Sampling

NeurIPS 2016poster

We unify slice sampling and Hamiltonian Monte Carlo (HMC) sampling, demonstrating their connection via the Hamiltonian-Jacobi equation from Hamiltonian mechanics. This insight enables extension of HMC and slice sampling to a broader family of samplers, called Monomial Gamma Samplers (MGS). We provid…

Cited by 24SourcePDFScholar
2015

On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators

NeurIPS 2015poster

Recent advances in Bayesian learning with large-scale data have witnessed emergence of stochastic gradient MCMC algorithms (SG-MCMC), such as stochastic gradient Langevin dynamics (SGLD), stochastic gradient Hamiltonian MCMC (SGHMC), and the stochastic gradient thermostat. While finite-time converge…

Cited by 214SourcePDFScholar
2015

Scalable Deep Poisson Factor Analysis for Topic Modeling

ICML 2015poster

A new framework for topic modeling is developed, based on deep graphical models, where interactions between topics are inferred through deep latent binary hierarchies. The proposed multi-layer model employs a deep sigmoid belief network or restricted Boltzmann machine, the bottom binary layer of whi…

Cited by 111SourcePDFScholar