AIConfPaper
← Search

Pierre Menard

24 accepted papers

2026

Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments

Romain Froger, Amine Benhalloum, Andrey Rusakov, Dheeraj Mekala, Emilien Garreau, Gerard Moreno-Torres Bertran +18
ICLR 2026oral

We introduce **Gaia2**, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal cons…

Cited by 0Source ↗Scholar ↗
2025

The Harder Path: Last Iterate Convergence for Uncoupled Learning in Zero-Sum Games with Bandit Feedback

Côme Fiegel, Pierre Menard, Tadashi Kozuno, Michal Valko, Vianney Perchet
ICML 2025poster

We study the problem of learning in zero-sum matrix games with repeated play and bandit feedback. Specifically, we focus on developing uncoupled algorithms that guarantee, without communication between players, convergence of the last-iterate to a Nash equilibrium. Although the non-bandit case has…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

Demonstration-Regularized RL

Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Alexey Naumov, Pierre Perrault +2
ICLR 2024poster

Incorporating expert demonstrations has empirically helped to improve the sample efficiency of reinforcement learning (RL). This paper quantifies theoretically to what extent this extra information reduces RL's sample complexity. In particular, we study the demonstration-regularized reinforcement le…

Cited by 0Source ↗PDF ↗Scholar ↗
2024

Local and Adaptive Mirror Descents in Extensive-Form Games

Côme Fiegel, Pierre Menard, Tadashi Kozuno, Remi Munos, Vianney Perchet, Michal Valko
NeurIPS 2024poster

We study how to learn $\epsilon$-optimal strategies in zero-sum imperfect information games (IIG) with *trajectory feedback*. In this setting, players update their policies sequentially, based on their observations over a fixed number of episodes denoted by $T$. Most existing procedures suffer from…

Cited by 3Source ↗PDF ↗Scholar ↗
2023

Adapting to game trees in zero-sum imperfect information games

Côme Fiegel, Pierre MENARD, Tadashi Kozuno, Remi Munos, Vianney Perchet, Michal Valko
ICML 2023oral

Imperfect information games (IIG) are games in which each player only partially observes the current game state. We study how to learn $\epsilon$-optimal strategies in a zero-sum IIG through self-play with trajectory feedback. We give a problem-independent lower bound $\widetilde{\mathcal{O}}(H(A_{\…

Cited by 16Source ↗PDF ↗code ↗Scholar ↗
2023

Fast Rates for Maximum Entropy Exploration

Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov +4
ICML 2023poster

We address the challenge of exploration in reinforcement learning (RL) when the agent operates in an unknown environment with sparse or no rewards. In this work, we study the maximum entropy exploration problem of two different types. The first type is visitation entropy maximization previously cons…

Cited by 22Source ↗PDF ↗code ↗Scholar ↗
2023

Model-free Posterior Sampling via Learning Rate Randomization

Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov +3
NeurIPS 2023poster

In this paper, we introduce Randomized Q-learning (RandQL), a novel randomized model-free algorithm for regret minimization in episodic Markov Decision Processes (MDPs). To the best of our knowledge, RandQL is the first tractable model-free posterior sampling-based algorithm. We analyze the performa…

Cited by 3Source ↗PDF ↗Scholar ↗
2023

Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice

Toshinori Kitamura, Tadashi Kozuno, Yunhao Tang, Nino Vieillard, Michal Valko, Wenhao Yang +9
ICML 2023poster

Mirror descent value iteration (MDVI), an abstraction of Kullback-Leibler (KL) and entropy-regularized reinforcement learning (RL), has served as the basis for recent high-performing practical RL algorithms. However, despite the use of function approximation in practice, the theoretical understandin…

Cited by 1Source ↗PDF ↗code ↗Scholar ↗
2022

Adaptive Multi-Goal Exploration

Jean Tarbouriech, Omar Darwiche Domingues, Pierre Menard, Matteo Pirotta, Michal Valko, Alessandro Lazaric
AISTATS 2022poster

We introduce a generic strategy for provably efficient multi-goal exploration. It relies on AdaGoal, a novel goal selection scheme that leverages a measure of uncertainty in reaching states to adaptively target goals that are neither too difficult nor too easy. We show how AdaGoal can be used to tac…

Cited by 5Source ↗PDF ↗Scholar ↗
2022

From Dirichlet to Rubin: Optimistic Exploration in RL without Bonuses

Daniil Tiapkin, Denis Belomestny, Eric Moulines, Alexey Naumov, Sergey Samsonov, Yunhao Tang +2
ICML 2022oral

We propose the Bayes-UCBVI algorithm for reinforcement learning in tabular, stage-dependent, episodic Markov decision process: a natural extension of the Bayes-UCB algorithm by Kaufmann et al. 2012 for multi-armed bandits. Our method uses the quantile of a Q-value function posterior as upper confide…

Cited by 24Source ↗PDF ↗Scholar ↗
2022

Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guarantees

Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov +3
NeurIPS 2022accept

We consider reinforcement learning in an environment modeled by an episodic, tabular, step-dependent Markov decision process of horizon $H$ with $S$ states, and $A$ actions. The performance of an agent is measured by the regret after interacting with the environment for $T$ episodes. We propose an…

Cited by 11Source ↗PDF ↗code ↗Scholar ↗
2021

A Kernel-Based Approach to Non-Stationary Reinforcement Learning in Metric Spaces

Omar Darwiche Domingues, Pierre Menard, Matteo Pirotta, Emilie Kaufmann, Michal Valko
AISTATS 2021poster

In this work, we propose KeRNS: an algorithm for episodic reinforcement learning in non-stationary Markov Decision Processes (MDPs) whose state-action set is endowed with a metric. Using a non-parametric model of the MDP built with time-dependent kernels, we prove a regret bound that scales with the…

Cited by 48Source ↗PDF ↗Scholar ↗
2021

Fast active learning for pure exploration in reinforcement learning

Pierre Menard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, Michal Valko
ICML 2021spotlight

Realistic environments often provide agents with very limited feedback. When the environment is initially unknown, the feedback, in the beginning, can be completely absent, and the agents may first choose to devote all their effort on \emph{exploring efficiently.} The exploration remains a challenge…

Cited by 97Source ↗PDF ↗code ↗Scholar ↗
2021

Indexed Minimum Empirical Divergence for Unimodal Bandits

Hassan SABER, Pierre MENARD, Odalric-Ambrym Maillard
NeurIPS 2021poster

We consider a stochastic multi-armed bandit problem specified by a set of one-dimensional family exponential distributions endowed with a unimodal structure. The unimodal structure is of practical relevance for several applications. We introduce IMED-UB, an algorithm that exploits provably optimally…

Cited by 6Source ↗PDF ↗Scholar ↗
2021

Kernel-Based Reinforcement Learning: A Finite-Time Analysis

Omar Darwiche Domingues, Pierre Menard, Matteo Pirotta, Emilie Kaufmann, Michal Valko
ICML 2021spotlight

We consider the exploration-exploitation dilemma in finite-horizon reinforcement learning problems whose state-action space is endowed with a metric. We introduce Kernel-UCBVI, a model-based optimistic algorithm that leverages the smoothness of the MDP and a non-parametric kernel estimator of the re…

Cited by 31Source ↗PDF ↗code ↗Scholar ↗
2021

Learning in two-player zero-sum partially observable Markov games with perfect recall

Tadashi Kozuno, Pierre MENARD, Remi Munos, Michal Valko
NeurIPS 2021poster

We study the problem of learning a Nash equilibrium (NE) in an extensive game with imperfect information (EGII) through self-play. Precisely, we focus on two-player, zero-sum, episodic, tabular EGII under the \textit{perfect-recall} assumption where the only feedback is realizations of the game (ban…

Cited by 51Source ↗PDF ↗Scholar ↗
2021

Problem Dependent View on Structured Thresholding Bandit Problems

James Cheshire, Pierre Menard, Alexandra Carpentier
ICML 2021spotlight

We investigate the \textit{problem dependent regime} in the stochastic \emph{Thresholding Bandit problem} (\tbp) under several \emph{shape constraints}. In the \tbp the objective of the learner is to output, after interacting with the environment, the set of arms whose means are above a given thresh…

Cited by 10Source ↗PDF ↗Scholar ↗
2021

UCB Momentum Q-learning: Correcting the bias without forgetting

Pierre Menard, Omar Darwiche Domingues, Xuedong Shang, Michal Valko
ICML 2021oral

We propose UCBMQ, Upper Confidence Bound Momentum Q-learning, a new algorithm for reinforcement learning in tabular and possibly stage-dependent, episodic Markov decision process. UCBMQ is based on Q-learning where we add a momentum term and rely on the principle of optimism in face of uncertainty t…

Cited by 50Source ↗PDF ↗code ↗Scholar ↗
2020

A single algorithm for both restless and rested rotting bandits

Julien Seznec, Pierre Menard, Alessandro Lazaric, Michal Valko
AISTATS 2020poster

In many application domains (e.g., recommender systems, intelligent tutoring systems), the rewards associated to the available actions tend to decrease over time. This decay is either caused by the actions executed in the past (e.g., a user may get bored when songs of the same genre are recommended…

Cited by 43Source ↗PDF ↗code ↗Scholar ↗
2020

Fixed-confidence guarantees for Bayesian best-arm identification

Xuedong Shang, Rianne Heide, Pierre Menard, Emilie Kaufmann, Michal Valko
AISTATS 2020poster

We investigate and provide new insights on the sampling rule called Top-Two Thompson Sampling (TTTS). In particular, we justify its use for fixed-confidence best-arm identification. We further propose a variant of TTTS called Top-Two Transportation Cost (T3C), which disposes of the computational bur…

Cited by 83Source ↗PDF ↗Scholar ↗
2020

Planning in Markov Decision Processes with Gap-Dependent Sample Complexity

Anders Jonsson, Emilie Kaufmann, Pierre Menard, Omar Darwiche Domingues, Edouard Leurent, Michal Valko
NeurIPS 2020poster

We propose MDP-GapE, a new trajectory-based Monte-Carlo Tree Search algorithm for planning in a Markov Decision Process in which transitions have a finite support. We prove an upper bound on the number of sampled trajectories needed for MDP-GapE to identify a near-optimal action with high probabilit…

Cited by 46Source ↗PDF ↗Scholar ↗
2019

Planning in entropy-regularized Markov decision processes and games

Jean-Bastien Grill, Omar Darwiche Domingues, Pierre Menard, Remi Munos, Michal Valko
NeurIPS 2019poster

We propose SmoothCruiser, a new planning algorithm for estimating the value function in entropy-regularized Markov decision processes and two-player games, given a generative model of the SmoothCruiser. SmoothCruiser makes use of the smoothness of the Bellman operator promoted by the regularization…

Cited by 20Source ↗PDF ↗code ↗Scholar ↗