Francesco Croce — accepted AI-conference papers

2026

Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning

Sara Ghazanfari, Francesco Croce, Nicolas Flammarion, Prashanth Krishnamurthy, Farshad Khorrami, Siddharth Garg

CVPR 2026

Recent work has shown that eliciting Large Language Models (LLMs) to generate reasoning traces in natural language before answering the user's request can significantly improve their performance across tasks. This approach has been extended to multimodal LLMs, where the models can produce chains-of-

Cited by 0 Source ↗code ↗Scholar ↗

2025

Is In-Context Learning Sufficient for Instruction Following in LLMs?

Hao Zhao, Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion

ICLR 2025poster

In-context learning (ICL) allows LLMs to learn from examples without changing their weights: this is a particularly promising capability for long-context LLMs that can potentially learn from many examples. Recently, Lin et al. (2024) proposed URIAL, a method using only three in-context examples to a…

Cited by 14 Source ↗PDF ↗code ↗Scholar ↗

2025

Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks

Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion

ICLR 2025poster

We show that even the most recent safety-aligned LLMs are not robust to simple *adaptive* jailbreaking attacks. First, we demonstrate how to successfully leverage access to *logprobs* for jailbreaking: we initially design an adversarial prompt template (sometimes adapted to the target LLM), and then…

Cited by 161 Source ↗PDF ↗code ↗Scholar ↗

2025

OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents

Thomas Kuntz, Agatha Duzan, Hao Zhao, Francesco Croce, J Zico Kolter, Nicolas Flammarion +1

NeurIPS 2025spotlight

Computer use agents are LLM-based agents that can directly interact with a graphical user interface, by processing screenshots or accessibility trees. While these systems are gaining popularity, their safety has been largely overlooked, despite the fact that evaluating and understanding their potent…

Cited by 0 Source ↗code ↗Scholar ↗

2025

Selective Induction Heads: How Transformers Select Causal Structures in Context

Francesco D'Angelo, Francesco Croce, Nicolas Flammarion

ICLR 2025poster

Transformers have exhibited exceptional capabilities in sequence modelling tasks, leveraging self-attention and in-context learning. Critical to this success are induction heads, attention circuits that enable copying tokens based on their previous occurrences. In this work, we introduce a novel syn…

Cited by 0 Source ↗PDF ↗Scholar ↗

2024

JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag +6

NeurIPS 2024poster

Jailbreak attacks cause large language models (LLMs) to generate harmful, unethical, or otherwise objectionable content. Evaluating these attacks presents a number of challenges, which the current collection of benchmarks and evaluation techniques do not adequately address. First, there is no clear…

Cited by 149 Source ↗PDF ↗code ↗Scholar ↗

2024

Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning

Hao Zhao, Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion

ICML 2024poster

There is a consensus that instruction fine-tuning of LLMs requires high-quality data, but what are they? LIMA (NeurIPS 2023) and AlpaGasus (ICLR 2024) are state-of-the-art methods for selecting such high-quality examples, either via manual curation or using GPT-3.5-Turbo as a quality scorer. We show…

Cited by 39 Source ↗PDF ↗code ↗Scholar ↗

2024

Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

Christian Schlarmann, Naman Deep Singh, Francesco Croce, Matthias Hein

ICML 2024oral

Multi-modal foundation models like OpenFlamingo, LLaVA, and GPT-4 are increasingly used for various real-world tasks. Prior work has shown that these models are highly vulnerable to adversarial attacks on the vision modality. These attacks can be leveraged to spread fake information or defraud users…

Cited by 50 Source ↗PDF ↗code ↗Scholar ↗

2023

A Modern Look at the Relationship between Sharpness and Generalization

Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, Nicolas Flammarion

ICML 2023poster

Sharpness of minima is a promising quantity that can correlate with generalization in deep networks and, when optimized during training, can improve generalization. However, standard sharpness is not invariant under reparametrizations of neural networks, and, to fix this, reparametrization-invariant…

Cited by 71 Source ↗PDF ↗code ↗Scholar ↗

2023

Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models

Naman Deep Singh, Francesco Croce, Matthias Hein

NeurIPS 2023poster

While adversarial training has been extensively studied for ResNet architectures and low resolution datasets like CIFAR-10, much less is known for ImageNet. Given the recent debate about whether transformers are more robust than convnets, we revisit adversarial training on ImageNet comparing ViTs an…

Cited by 69 Source ↗PDF ↗code ↗Scholar ↗

2023

Revisiting adapters with adversarial training

Sylvestre-Alvise Rebuffi, Francesco Croce, Sven Gowal

ICLR 2023top-25%

While adversarial training is generally used as a defense mechanism, recent works show that it can also act as a regularizer. By co-training a neural network on clean and adversarial inputs, it is possible to improve classification accuracy on the clean, non-adversarial inputs. We demonstrate that,…

Cited by 18 Source ↗PDF ↗Scholar ↗

2023

Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shifts

Francesco Croce, Sylvestre-Alvise Rebuffi, Evan Shelhamer, Sven Gowal

CVPR 2023poster

Adversarial training is widely used to make classifiers robust to a specific threat or adversary, such as l_p-norm bounded perturbations of a given p-norm. However, existing methods for training classifiers robust to multiple threats require knowledge of all attacks during training and remain vulner…

Cited by 21 Source ↗PDF ↗Scholar ↗

2022

Adversarial Robustness against Multiple and Single $l_p$-Threat Models via Quick Fine-Tuning of Robust Classifiers

Francesco Croce, Matthias Hein

ICML 2022spotlight

A major drawback of adversarially robust models, in particular for large scale datasets like ImageNet, is the extremely long training time compared to standard models. Moreover, models should be robust not only to one $l_p$-threat model but ideally to all of them. In this paper we propose Extreme no…

Cited by 22 Source ↗PDF ↗code ↗Scholar ↗

2022

Diffusion Visual Counterfactual Explanations

Maximilian Augustin, Valentyn Boreiko, Francesco Croce, Matthias Hein

NeurIPS 2022accept

Visual Counterfactual Explanations (VCEs) are an important tool to understand the decisions of an image classifier. They are “small” but “realistic” semantic changes of the image changing the classifier decision. Current approaches for the generation of VCEs are restricted to adversarially robust mo…

Cited by 81 Source ↗PDF ↗code ↗Scholar ↗

2022

Evaluating the Adversarial Robustness of Adaptive Test-time Defenses

Francesco Croce, Sven Gowal, Thomas Brunner, Evan Shelhamer, Matthias Hein, Taylan Cemgil

ICML 2022spotlight

Adaptive defenses, which optimize at test time, promise to improve adversarial robustness. We categorize such adaptive test-time defenses, explain their potential benefits and drawbacks, and evaluate a representative variety of the latest adaptive defenses for image classification. Unfortunately, no…

Cited by 89 Source ↗PDF ↗code ↗Scholar ↗

2022

Sparse-RS: A Versatile Framework for Query-Efficient Sparse Black-Box Adversarial Attacks

Francesco Croce, Maksym Andriushchenko, Naman D. Singh, Nicolas Flammarion, Matthias Hein

AAAI 2022technical

We propose a versatile framework based on random search, Sparse-RS, for score-based sparse targeted and untargeted attacks in the black-box setting. Sparse-RS does not rely on substitute models and achieves state-of-the-art success rate and query efficiency for multiple sparse attack models: L0-boun…

Cited by 129 Source ↗PDF ↗code ↗Scholar ↗

2021

Mind the Box: $l_1$-APGD for Sparse Adversarial Attacks on Image Classifiers

Francesco Croce, Matthias Hein

ICML 2021spotlight

We show that when taking into account also the image domain $[0,1]^d$, established $l_1$-projected gradient descent (PGD) attacks are suboptimal as they do not consider that the effective threat model is the intersection of the $l_1$-ball and $[0,1]^d$. We study the expected sparsity of the steepest…

Cited by 77 Source ↗PDF ↗Scholar ↗

2021

RobustBench: a standardized adversarial robustness benchmark

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang +2

NeurIPS 2021poster

As a research community, we are still lacking a systematic understanding of the progress on adversarial robustness which often makes it hard to identify the most promising ideas in training robust models. A key challenge in benchmarking robustness is that its evaluation is often error-prone leading…

Cited by 843 Source ↗code ↗Scholar ↗

2020

Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack

Francesco Croce, Matthias Hein

ICML 2020poster

The evaluation of robustness against adversarial manipulation of neural networks-based classifiers is mainly tested with empirical attacks as methods for the exact computation, even when available, do not scale to large networks. We propose in this paper a new white-box adversarial attack wrt the $l…

Cited by 607 Source ↗PDF ↗code ↗Scholar ↗

2020

Provable robustness against all adversarial $l_p$-perturbations for $p\geq 1$

Francesco Croce, Matthias Hein

ICLR 2020poster

In recent years several adversarial attacks and defenses have been proposed. Often seemingly robust models turn out to be non-robust when more sophisticated attacks are used. One way out of this dilemma are provable robustness guarantees. While provably robust models for specific $l_p$-perturbation…

Cited by 6 Source ↗code ↗Scholar ↗

2020

Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

Francesco Croce, Matthias Hein

ICML 2020poster

The field of defense strategies against adversarial attacks has significantly grown over the last years, but progress is hampered as the evaluation of adversarial defenses is often insufficient and thus gives a wrong impression of robustness. Many promising defenses could be broken later on, making…

Cited by 2290 Source ↗PDF ↗code ↗Scholar ↗

2020

Square Attack: a query-efficient black-box adversarial attack via random search

Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, Matthias Hein

ECCV 2020poster

We propose the Square Attack, a score-based black-box $l_2$- and $l_\infty$- adversarial attack that does not rely on local gradient information and thus is not affected by gradient masking. Square Attack is based on a randomized search scheme which selects localized square-shaped updates at random…

Cited by 1243 Source ↗PDF ↗code ↗Scholar ↗

2019

Provable Robustness of ReLU networks via Maximization of Linear Regions

Francesco Croce, Maksym Andriushchenko, Matthias Hein

AISTATS 2019poster

It has been shown that neural network classifiers are not robust. This raises concerns about their usage in safety-critical systems. We propose in this paper a regularization scheme for ReLU networks which provably improves the robustness of the classifier by maximizing the linear regions of the cla…

Cited by 195 Source ↗PDF ↗code ↗Scholar ↗

2019

Sparse and Imperceivable Adversarial Attacks

Francesco Croce, Matthias Hein

ICCV 2019poster

Neural networks have been proven to be vulnerable to a variety of adversarial attacks. From a safety perspective, highly sparse adversarial attacks are particularly dangerous. On the other hand the pixelwise perturbations of sparse attacks are typically large and thus can be potentially detected. We…

Cited by 259 PDF ↗code ↗Scholar ↗