← Search

Jonas Geiping

55 accepted papers

2026

Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols

ICLR 2026poster

AI control protocols serve as a defense mechanism to stop untrusted LLM agents from causing harm in autonomous settings. Prior work treats this as a security problem, stress testing with exploits that use the deployment context to subtly complete harmful side tasks, such as backdoor insertion. In pr…

Cited by 0SourceScholar
2026

Capability-Based Scaling Trends for LLM-Based Red-Teaming

ICLR 2026poster

As large language models grow in capability and agency, identifying vulnerabilities through red-teaming becomes vital for safe deployment. However, traditional prompt-engineering approaches may prove ineffective once red-teaming turns into a \emph{weak-to-strong} problem, where target models surpass…

Cited by 0SourcecodeScholar
2026

Curating the Future: A Scalable Recipe for Training Open-Ended Forecasters

ICML 2026poster

High-stakes decision making involves reasoning under uncertainty about the future. In this work, we train language models to make predictions on open-ended forecasting questions. To scale up training data, we synthesize novel forecasting questions from global events reported in daily news. While dir…

Cited by 0SourceScholar
2026

GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching

ICLR 2026poster

Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in deployment and inference. While structured pruning of model parameters…

Cited by 0SourcecodeScholar
2026

MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation

AAAI 2026technical

Universal medical image segmentation models have emerged as a promising paradigm due to their strong generalizability across diverse tasks, showing great potential for a wide range of clinical applications. This potential has been partly driven by the success of general-purpose vision models such as

Cited by 0SourcePDFScholar
2026

Rewiring Experts on the Fly: Continuous Rerouting for Better Online Adaptation in Mixture-of-Expert models

ICML 2026poster

Mixture-of-Experts (MoE) models achieve efficient scaling through sparse expert activation, but often suffer from suboptimal routing decisions due to distribution shifts in deployment. While existing test-time adaptation methods could potentially address these issues, they primarily focus on dense m…

Cited by 0SourceScholar
2026

Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs

ICLR 2026poster

Large Language Models (LLMs) are increasingly applied to complex tasks that require extended reasoning. In such settings, models often benefit from diverse chains-of-thought to arrive at multiple candidate solutions. This requires two competing objectives: to inject enough stochasticity to explore m…

Cited by 0SourceScholar
2026

Strategic Dishonesty Can Undermine AI Safety Evaluations of Frontier LLMs

ICLR 2026poster

Large language model (LLM) developers aim for their models to be honest, helpful, and harmless. However, when faced with malicious requests, models are trained to refuse, sacrificing helpfulness. We show that frontier LLMs can develop a preference for \textit{dishonesty} as a new strategy, even when…

Cited by 0SourceScholar
2026

The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs

ICLR 2026poster

Does continued scaling of large language models (LLMs) yield diminishing returns? In this work, we show that short-task benchmarks may give an illusion of slowing progress, as even marginal gains in single-step accuracy can compound into exponential improvements in the length of tasks a model can su…

Cited by 0SourcecodeScholar
2026

Training AI Co-Scientists Using Rubric Rewards

ICML 2026poster

AI co-scientists are emerging as a useful tool for human researchers, with a crucial ability being proposing a research plan for a given research goal. In this work, we study how to train language models that generate better research plans by leveraging the vast corpus of existing research papers. T…

Cited by 0SourceScholar
2025

An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks

ICML 2025poster

A plethora of jailbreaking attacks have been proposed to obtain harmful responses from safety-tuned LLMs. These methods largely succeed in coercing the target output in their original settings, but their attacks vary substantially in fluency and computational effort. In this work, we propose a unifi…

Cited by 0SourcePDFScholar
2025

Great Models Think Alike and this Undermines AI Oversight

ICML 2025spotlight

As Language Model (LM) capabilities advance, evaluating and supervising them at scale is getting harder for humans. There is hope that other language models can automate both these tasks, which we refer to as *AI Oversight*. We study how model similarity affects both aspects of AI oversight by propo…

2025

LLM-Generated Passphrases That Are Secure and Easy to Remember

NAACL 2025findings

Automatically generated passwords and passphrases are a cornerstone of IT security. Yet, these passphrases are often hard to remember and see only limited adoption. In this work, we use large language models to generate passphrases with rigorous security guarantees via the computation of the entropy…

2025

Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

NeurIPS 2025spotlight

We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale…

Cited by 0SourcecodeScholar
2024

Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs

NeurIPS 2024poster

Large language models can memorize and repeat their training data, causing privacy and copyright risks. To mitigate memorization, we introduce a subtle modification to the next-token training objective that we call the goldfish loss. During training, a randomly sampled subsets of tokens are excluded…

2024

CALVIN: Improved Contextual Video Captioning via Instruction Tuning

NeurIPS 2024poster

The recent emergence of powerful Vision-Language models (VLMs) has significantly improved image captioning. Some of these models are extended to caption videos as well. However, their capabilities to understand complex scenes are limited, and the descriptions they provide for scenes tend to be overl…

Cited by 0SourcePDFScholar
2024

NEFTune: Noisy Embeddings Improve Instruction Finetuning

ICLR 2024poster

We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard finetuning of LLaMA-2-7B using Alpaca achieves $29.79$\% on AlpacaEval, which rises to $64.69$\% using noisy embeddings.…

2024

Object Recognition as Next Token Prediction

CVPR 2024highlight

We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels. To ground this prediction process in auto-regression we customize a non-causal attention mask for…

2024

On the Reliability of Watermarks for Large Language Models

ICLR 2024poster

As LLMs become commonplace, machine-generated text has the potential to flood the internet with spam, social media bots, and valueless content. _Watermarking_ is a simple and effective strategy for mitigating such harms by enabling the detection and documentation of LLM-generated text. Yet a crucial…

2024

Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models

NeurIPS 2024poster

It is commonplace to produce application-specific models by fine-tuning large pre-trained models using a small bespoke dataset. The widespread availability of foundation model checkpoints on the web poses considerable risks, including the vulnerability to backdoor attacks. In this paper, we unveil a…

Cited by 18SourcePDFScholar
2024

Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

ICML 2024poster

Detecting text generated by modern large language models is thought to be hard, as both LLMs and humans can exhibit a wide range of complex behaviors. However, we find that a score based on contrasting two closely related language models is highly accurate at separating human-generated and machine-g…

2024

Transformers Can Do Arithmetic with the Right Embeddings

NeurIPS 2024poster

The poor performance of transformers on arithmetic tasks seems to stem in large part from their inability to keep track of the exact position of each digit inside of a large span of digits. We mend this problem by adding an embedding to each digit that encodes its position relative to the start of t…

2024

Universal Guidance for Diffusion Models

ICLR 2024poster

Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalit…

2023

A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning

NeurIPS 2023poster

Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even engineer new features from existing ones. To prevent over-fitting in subsequent downstream modeling, practitioners commo…

2023

A Watermark for Large Language Models

ICML 2023oral

Potential harms of large language models can be mitigated by watermarking model output, i.e., embedding signals into generated text that are invisible to humans but algorithmically detectable from a short span of tokens. We propose a watermarking framework for proprietary language models. The waterm…

2023

Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries

ICLR 2023top-25%

As industrial applications are increasingly automated by machine learning models, enforcing personal data ownership and intellectual property rights requires tracing training data back to their rightful owners. Membership inference algorithms approach this problem by using statistical techniques to…

2023

Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise

NeurIPS 2023poster

Standard diffusion models involve an image transform -- adding Gaussian noise -- and an image restoration operator that inverts this degradation. We observe that the generative behavior of diffusion models is not strongly dependent on the choice of image degradation, and in fact, an entire family…

2023

Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models

ICLR 2023poster

Privacy is a central tenet of Federated learning (FL), in which a central server trains models without centralizing user data. However, gradient updates used in FL can leak user information. While the most industrial uses of FL are for text applications (e.g. keystroke prediction), the majority of…

2023

Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models

CVPR 2023poster

Cutting-edge diffusion models produce images with high quality and customizability, enabling them to be used for commercial art and graphic design purposes. But do diffusion models create unique works of art, or are they replicating content directly from their training sets? In this work, we study i…

2023

Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

NeurIPS 2023poster

The strength of modern generative models lies in their ability to be controlled through prompts. Hard prompts comprise interpretable words and tokens, and are typically hand-crafted by humans. Soft prompts, on the other hand, consist of continuous feature vectors. These can be discovered using pow…

2023

How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularization

ICLR 2023poster

Despite the clear performance benefits of data augmentations, little is known about why they are so effective. In this paper, we disentangle several key mechanisms through which data augmentations operate. Establishing an exchange rate between augmented and additional real data, we find that in out-…

2023

Loss Landscapes are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent

ICLR 2023top-25%

It is commonly believed that the implicit regularization of optimizers is needed for neural networks to generalize in the overparameterized regime. In this paper, we observe experimentally that this implicit regularization behavior is {\em generic}, i.e. it does not depend strongly on the choice of…

2023

On the Exploitability of Instruction Tuning

NeurIPS 2023poster

Instruction tuning is an effective technique to align large language models (LLMs) with human intent. In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavi…

2023

Panning for Gold in Federated Learning: Targeted Text Extraction under Arbitrarily Large-Scale Aggregation

ICLR 2023poster

As federated learning (FL) matures, privacy attacks against FL systems in turn become more numerous and complex. Attacks on language models have progressed from recovering single sentences in simple classification tasks to recovering larger parts of user data. Current attacks against federated langu…

Cited by 13SourcePDFScholar
2023

STYX: Adaptive Poisoning Attacks Against Byzantine-Robust Defenses in Federated Learning

ICASSP 2023accepted

Decentralized training of machine learning models, for instance with federated learning protocols, continues to diffuse from theory toward practical applications and use cases. In federated learning (FL), a central server trains a model collaboratively with a group of users by communicating model up…

Cited by 0SourceScholar
2023

Tree-Rings Watermarks: Invisible Fingerprints for Diffusion Images

NeurIPS 2023poster

Watermarking the outputs of generative models is a crucial technique for tracing copyright and preventing potential harm from AI-generated content. In this paper, we introduce a novel technique called Tree-Ring Watermarking that robustly fingerprints diffusion model outputs. Unlike existing methods…

Cited by 60SourcePDFScholar
2023

Understanding and Mitigating Copying in Diffusion Models

NeurIPS 2023poster

Images generated by diffusion models like Stable Diffusion are increasingly widespread. Recent works and even lawsuits have shown that these models are prone to replicating their training data, unbeknownst to the user. In this paper, we first analyze this memorization problem in text-to-image diffus…

2023

What Can We Learn from Unlearnable Datasets?

NeurIPS 2023poster

In an era of widespread web scraping, unlearnable dataset methods have the potential to protect data privacy by preventing deep neural networks from generalizing. But in addition to a number of practical limitations that make their use unlikely, we make a number of findings that call into question t…

2022

Autoregressive Perturbations for Data Poisoning

NeurIPS 2022accept

The prevalence of data scraping from social media as a means to obtain datasets has led to growing concerns regarding unauthorized use of data. Data poisoning attacks have been proposed as a bulwark against scraping, as they make data ``unlearnable'' by adding small, imperceptible perturbations. Unf…

2022

Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models

ICLR 2022poster

Federated learning has quickly gained popularity with its promises of increased user privacy and efficiency. Previous works have shown that federated gradient updates contain information that can be used to approximately recover user data in some situations. These previous attacks on user privacy…

2022

Stochastic Training is Not Necessary for Generalization

ICLR 2022poster

It is widely believed that the implicit regularization of SGD is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve comparably strong performance to SGD on CIFAR-10 using modern archit…

2021

Adversarial Examples Make Strong Poisons

NeurIPS 2021poster

The adversarial machine learning literature is largely partitioned into evasion attacks on testing data and poisoning attacks on training data. In this work, we show that adversarial examples, originally intended for attacking pre-trained models, are even more effective for data poisoning than rece…

2021

Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradeoff

ICASSP 2021accepted

Data poisoning and backdoor attacks manipulate victim models by maliciously modifying training data. In light of this growing threat, a recent survey of industry professionals revealed heightened fear in the private sector regarding data poisoning. Many previous defenses against poisoning either fai…

Cited by 0SourceScholar
2021

Witches' Brew: Industrial Scale Data Poisoning via Gradient Matching

ICLR 2021poster

Data Poisoning attacks modify training data to maliciously control a model trained on such data. In this work, we focus on targeted poisoning attacks which cause a reclassification of an unmodified test image and as such breach model integrity. We consider a particularly malicious poisoning attack t…

2020

Inverting Gradients - How easy is it to break privacy in federated learning?

NeurIPS 2020poster

The idea of federated learning is to collaboratively train a neural network on a server. Each user receives the current weights of the network and in turns sends parameter updates (gradients) based on local data. This protocol has been designed not only to train neural networks data-efficiently, but…

2020

MetaPoison: Practical General-purpose Clean-label Data Poisoning

NeurIPS 2020poster

Data poisoning---the process by which an attacker takes control of a model by making imperceptible changes to a subset of the training data---is an emerging threat in the context of neural networks. Existing attacks for data poisoning neural networks have relied on hand-crafted heuristics, because s…

2020

Truth or backpropaganda? An empirical investigation of deep learning theory

ICLR 2020spotlight

We empirically evaluate common assumptions about neural networks that are widely held by practitioners and theorists alike. In this work, we: (1) prove the widespread existence of suboptimal local minima in the loss landscape of neural networks, and we use our theory to find examples; (2) show that…

Cited by 46SourcecodeScholar
2020

Witchcraft: Efficient PGD Attacks with Random Step Size

ICASSP 2020accepted

State-of-the-art adversarial attacks on neural networks use expensive iterative methods and numerous random restarts from different initial points. Iterative FGSM-based methods without restarts trade off performance for computational efficiency because they do not adequately explore the image space…

Cited by 0SourceScholar