Maja Pantic — accepted AI-conference papers

2026

FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANs

Andreas Zinonos, Michał Stypułkowski, Antoni Bigata, Stavros Petridis, Maja Pantic, Nikita Drobyshev

CVPR 2026

We present FlashLips, a two-stage, mask-free lip-sync system that decouples lips control from rendering and achieves real-time performance, with our U-Net variant running at over 100 FPS on a single GPU, while matching the visual quality of larger state-of-the-art models. Stage 1 is a compact, one-s

Cited by 0 Source ↗Scholar ↗

2025

Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extraction

Minsu Kim, Rodrigo Mira, Honglie Chen, Stavros Petridis, Maja Pantic

ICASSP 2025accepted

In this paper, we investigate a novel approach for Target Speech Extraction (TSE), which relies solely on textual context to extract the target speech. We refer to this task as Contextual Speech Extraction (CSE). Unlike traditional TSE methods that rely on pre-recorded enrollment utterances, video o…

Cited by 0 Source ↗Scholar ↗

2025

Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models

Adriana Fernandez-Lopez, Shiwei Liu, Lu Yin, Stavros Petridis, Maja Pantic

ICASSP 2025accepted

This paper investigates the under-explored area of low-rank weight training for large-scale Conformer-based speech recognition models from scratch. Our study demonstrates the viability of this training paradigm for such models, yielding several notable findings. Firstly, we discover that applying a…

Cited by 0 Source ↗Scholar ↗

2025

KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation

Antoni Bigata, Michał Stypułkowski, Rodrigo Mira, Stella Bounareli, Konstantinos Vougioukas, Zoe Landgraf +4

CVPR 2025poster

Current audio-driven facial animation methods achieve impressive results for short videos but suffer from error accumulation and identity drift when extended to longer durations. Existing methods attempt to mitigate this through external spatial control, increasing long-term consistency but compromi…

Cited by 2 Source ↗PDF ↗Scholar ↗

2025

Large Language Models are Strong Audio-Visual Speech Recognition Learners

Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna +2

ICASSP 2025accepted

Multimodal large language models (MLLMs) have recently become a focal point of research due to their formidable multimodal understanding capabilities. For example, in the audio and speech domains, an LLM can be equipped with (automatic) speech recognition (ASR) abilities by just concatenating the au…

Cited by 0 Source ↗Scholar ↗

2025

MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition

Umberto Cappellazzo, Minsu Kim, Pingchuan Ma, Honglie Chen, Xubo Liu, Stavros Petridis +1

NeurIPS 2025poster

Large language models (LLMs) have recently shown strong potential in audio-visual speech recognition (AVSR), but their high computational demands and sensitivity to token granularity limit their practicality in resource-constrained settings. Token compression methods can reduce inference cost, but t…

Cited by 0 Source ↗Scholar ↗

2024

BRAVEn: Improving Self-supervised pre-training for Visual and Auditory Speech Recognition

Alexandros Haliassos, Andreas Zinonos, Rodrigo Mira, Stavros Petridis, Maja Pantic

ICASSP 2024accepted

Self-supervision has recently shown great promise for learning visual and auditory speech representations from unlabelled data. In this work, we propose BRAVEn, an extension to the recent RAVEn method, which learns speech representations entirely from raw audio-visual data. Our modifications to RAVE…

Cited by 0 Source ↗Scholar ↗

2024

EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars

Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, Maja Pantic

CVPR 2024poster

Head avatars animated by visual signals have gained popularity particularly in cross-driving synthesis where the driver differs from the animated character a challenging but highly practical approach. The recently presented MegaPortraits model has demonstrated state-of-the-art results in this domain…

Cited by 26 Source ↗PDF ↗Scholar ↗

2024

Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs

Alexandros Haliassos, Rodrigo Mira, Honglie Chen, Zoe Landgraf, Stavros Petridis, Maja Pantic

NeurIPS 2024poster

Research in auditory, visual, and audiovisual speech recognition (ASR, VSR, and AVSR, respectively) has traditionally been conducted independently. Even recent self-supervised studies addressing two or all three tasks simultaneously tend to yield separate models, leading to disjoint inference pipeli…

Cited by 3 Source ↗PDF ↗code ↗Scholar ↗

2023

Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels

Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Lopez, Honglie Chen, Stavros Petridis, Maja Pantic

ICASSP 2023accepted

Audio-visual speech recognition has received a lot of attention due to its robustness against acoustic noise. Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger mode…

Cited by 0 Source ↗Scholar ↗

2023

Jointly Learning Visual and Auditory Speech Representations from Raw Data

Alexandros Haliassos, Pingchuan Ma, Rodrigo Mira, Stavros Petridis, Maja Pantic

ICLR 2023poster

We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets generated by slowly-evolving momentum encoders. Driven by the inherent differen…

Cited by 57 Source ↗PDF ↗code ↗Scholar ↗

2023

LA-VOCE: LOW-SNR Audio-Visual Speech Enhancement Using Neural Vocoders

Rodrigo Mira, Buye Xu, Jacob Donley, Anurag Kumar, Stavros Petridis, Vamsi Krishna Ithapu +1

ICASSP 2023accepted

Audio-visual speech enhancement aims to extract clean speech from a noisy environment by leveraging not only the audio itself but also the target speaker’s lip movements. This approach has been shown to yield improvements over audio-only speech enhancement, particularly for the removal of interferin…

Cited by 0 Source ↗Scholar ↗

2023

Learning Cross-Lingual Visual Speech Representations

Andreas Zinonos, Alexandros Haliassos, Pingchuan Ma, Stavros Petridis, Maja Pantic

ICASSP 2023accepted

Cross-lingual self-supervised learning has been a growing research topic in the last few years. However, current works only explored the use of audio signals to create representations. In this work, we study cross-lingual self-supervised visual representation learning. We use the recently-proposed R…

Cited by 0 Source ↗Scholar ↗

2023

SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision

Xubo Liu, Egor Lakomkin, Konstantinos Vougioukas, Pingchuan Ma, Honglie Chen, Ruiming Xie +6

CVPR 2023poster

Recently reported state-of-the-art results in visual speech recognition (VSR) often rely on increasingly large amounts of video data, while the publicly available transcribed video datasets are limited in size. In this paper, for the first time, we study the potential of leveraging synthetic visual…

Cited by 27 Source ↗PDF ↗Scholar ↗

2022

Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection

Alexandros Haliassos, Rodrigo Mira, Stavros Petridis, Maja Pantic

CVPR 2022poster

One of the most pressing challenges for the detection of face-manipulated videos is generalising to forgery methods not seen during training while remaining effective under common corruptions such as compression. In this paper, we examine whether we can tackle this issue by harnessing videos of real…

Cited by 149 PDF ↗code ↗Scholar ↗

2022

Training Strategies for Improved Lip-Reading

Pingchuan Ma, Yujiang Wang, Stavros Petridis, Jie Shen, Maja Pantic

ICASSP 2022accepted

Several training strategies and temporal models have been recently proposed for isolated word lip-reading in a series of independent works. However, the potential of combining the best strategies and investigating the impact of each of them has not been explored. In this paper, we systematically inv…

Cited by 0 Source ↗Scholar ↗

2021

DINO: A Conditional Energy-Based GAN for Domain Translation

Konstantinos Vougioukas, Stavros Petridis, Maja Pantic

ICLR 2021poster

Domain translation is the process of transforming data from one domain to another while preserving the common semantics. Some of the most popular domain translation systems are based on conditional generative adversarial networks, which use source domain data to drive the generator and as an input t…

Cited by 10 Source ↗PDF ↗code ↗Scholar ↗

2021

Detecting Adversarial Attacks on Audiovisual Speech Recognition

Pingchuan Ma, Stavros Petridis, Maja Pantic

ICASSP 2021accepted

Adversarial attacks pose a threat to deep learning models. However, research on adversarial detection methods, especially in the multi-modal domain, is very limited. In this work, we propose an efficient and straightforward detection method based on the temporal correlation between audio and video s…

Cited by 0 Source ↗Scholar ↗

2021

End-To-End Audio-Visual Speech Recognition with Conformers

Pingchuan Ma, Stavros Petridis, Maja Pantic

ICASSP 2021accepted

In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner. In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms, respectively…

Cited by 0 Source ↗Scholar ↗

2021

Learning Separable Time-Frequency Filterbanks for Audio Classification

Jie Pu, Yannis Panagakis, Maja Pantic

ICASSP 2021accepted

State-of-the-art audio classification systems often apply deep neural networks on hand-crafted features (e.g., spectrogram-based representations), instead of learning features directly from raw audio. Moreover, these audio networks have millions of unknown parameters need to be learned, which causes…

Cited by 0 Source ↗Scholar ↗

2021

Lips Don't Lie: A Generalisable and Robust Approach To Face Forgery Detection

Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, Maja Pantic

CVPR 2021poster

Although current deep learning-based face forgery detectors achieve impressive performance in constrained scenarios, they are vulnerable to samples created by unseen manipulation methods. Some recent works show improvements in generalisation but rely on cues that are easily corrupted by common post-…

Cited by 515 PDF ↗code ↗Scholar ↗

2021

Towards Practical Lipreading with Distilled and Efficient Models

Pingchuan Ma, Brais Martínez, Stavros Petridis, Maja Pantic

ICASSP 2021accepted

Lipreading has witnessed a lot of progress due to the resurgence of neural networks. Recent works have placed emphasis on aspects such as improving performance by finding the optimal architecture or improving generalization. However, there is still a significant gap between the current methodologies…

Cited by 0 Source ↗Scholar ↗

2020

Dynamic Face Video Segmentation via Reinforcement Learning

Yujiang Wang, Mingzhi Dong, Jie Shen, Yang Wu, Shiyang Cheng, Maja Pantic

CVPR 2020poster

For real-time semantic video segmentation, most recent works utilised a dynamic framework with a key scheduler to make online key/non-key decisions. Some works used a fixed key scheduling policy, while others proposed adaptive key scheduling methods based on heuristic strategies, both of which may l…

Cited by 31 PDF ↗Scholar ↗

2020

Factorized Higher-Order CNNs With an Application to Spatio-Temporal Emotion Estimation

Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M. Hospedales, Maja Pantic

CVPR 2020poster

Training deep neural networks with spatio-temporal (i.e., 3D) or multidimensional convolutions of higher-order is computationally challenging due to millions of unknown parameters across dozens of layers. To alleviate this, one approach is to apply low-rank tensor decompositions to convolution kerne…

Cited by 109 PDF ↗Scholar ↗

2020

Learning Differentiable Sparse and Low Rank Networks for Audio-Visual Object Localization

Jie Pu, Yannis Panagakis, Maja Pantic

ICASSP 2020accepted

Parsimonious modelling, including sparsity and low rankness, has becomes a cornerstone in modern machine learning and signal processing. However, these modelling techniques have limited capabity to learn from large-scale data, and often require some pre-defined parameters to define their optimizatio…

Cited by 0 Source ↗Scholar ↗

2020

Lipreading Using Temporal Convolutional Networks

Brais Martínez, Pingchuan Ma, Stavros Petridis, Maja Pantic

ICASSP 2020accepted

Lip-reading has attracted a lot of research attention lately thanks to advances in deep learning. The current state-of-the-art model for recognition of isolated words in-the-wild consists of a residual network and Bidirectional Gated Recurrent Unit (BGRU) layers. In this work, we address the limitat…

Cited by 0 Source ↗Scholar ↗

2020

Multilinear Latent Conditioning for Generating Unseen Attribute Combinations

Markos Georgopoulos, Grigorios Chrysos, Maja Pantic, Yannis Panagakis

ICML 2020poster

Deep generative models rely on their inductive bias to facilitate generalization, especially for problems with high dimensional data, like images. However, empirical studies have shown that variational autoencoders (VAE) and generative adversarial networks (GAN) lack the generalization ability that…

Cited by 17 Source ↗PDF ↗Scholar ↗

2020

Speech-Driven Facial Animation Using Polynomial Fusion of Features

Triantafyllos Kefalas, Konstantinos Vougioukas, Yannis Panagakis, Stavros Petridis, Jean Kossaifi, Maja Pantic

ICASSP 2020accepted

Speech-driven facial animation involves using a speech signal to generate realistic videos of talking faces. Recent deep learning approaches to facial synthesis rely on extracting low-dimensional representations and concatenating them, followed by a decoding step of the concatenated vector. This acc…

Cited by 0 Source ↗Scholar ↗

2020

Towards Pose-Invariant Lip-Reading

Shiyang Cheng, Pingchuan Ma, Georgios Tzimiropoulos, Stavros Petridis, Adrian Bulat, Jie Shen +1

ICASSP 2020accepted

Lip-reading models have been significantly improved recently thanks to powerful deep learning architectures. However, most works focused on frontal or near frontal views of the mouth. As a consequence, lip-reading performance seriously deteriorates in non-frontal mouth views. In this work, we presen…

Cited by 0 Source ↗Scholar ↗

2020

Visually Guided Self Supervised Learning of Speech Representations

Abhinav Shukla, Konstantinos Vougioukas, Pingchuan Ma, Stavros Petridis, Maja Pantic

ICASSP 2020accepted

Self supervised representation learning has recently attracted a lot of research interest for both the audio and visual modalities. However, most works typically focus on a particular modality or feature alone and there has been very limited work that studies the interaction between the two modaliti…

Cited by 0 Source ↗Scholar ↗

2019

Learning Low Rank and Sparse Models via Robust Autoencoders

Jie Pu, Yannis Panagakis, Maja Pantic

ICASSP 2019accepted

Robust principal component analysis (RPCA), decomposes a data matrix into a superposition of a low-rank matrix and a sparse matrix under certain incoherent conditions. In this paper, we propose a nonlinear generalization of RPCA that uses two autoencoder networks to achieve such a decomposition, in…

Cited by 0 Source ↗Scholar ↗

2019

T-Net: Parametrizing Fully Convolutional Nets With a Single High-Order Tensor

Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, Maja Pantic

CVPR 2019poster

Recent findings indicate that over-parametrization, while crucial for successfully training deep neural networks, also introduces large amounts of redundancy. Tensor methods have the potential to efficiently parametrize over-complete representations by leveraging this redundancy. In this paper, we p…

Cited by 93 PDF ↗Scholar ↗

2018

4DFAB: A Large Scale 4D Database for Facial Expression Analysis and Biometric Applications

Shiyang Cheng, Irene Kotsia, Maja Pantic, Stefanos Zafeiriou

CVPR 2018poster

The progress we are currently witnessing in many computer vision applications, including automatic face analysis, would not be made possible without tremendous efforts in collecting and annotating large scale visual databases. To this end, we propose 4DFAB, a new large scale database of dynamic high…

Cited by 137 Source ↗PDF ↗Scholar ↗

2018

End-to-End Audiovisual Speech Recognition

Stavros Petridis, Themos Stafylakis, Pingchuan Ma, Feipeng Cai, Georgios Tzimiropoulos, Maja Pantic

ICASSP 2018accepted

Several end-to-end deep learning approaches have been recently presented which extract either audio or visual features from the input images or audio signals and perform speech recognition. However, research on end-to-end audiovisual models is very limited. In this work, we present an end-to-end aud…

Cited by 0 Source ↗Scholar ↗

2018

GAGAN: Geometry-Aware Generative Adversarial Networks

Jean Kossaifi, Linh Tran, Yannis Panagakis, Maja Pantic

CVPR 2018poster

Deep generative models learned through adversarial training have become increasingly popular for their ability to generate naturalistic image textures. However, aside from their texture, the visual appearance of objects is significantly influenced by their shape geometry; information which is not ta…

Cited by 69 Source ↗PDF ↗Scholar ↗

2018

Visual-Only Recognition of Normal, Whispered and Silent Speech

Stavros Petridis, Jie Shen, Doruk Cetin, Maja Pantic

ICASSP 2018accepted

Silent speech interfaces have been recently proposed as a way to enable communication when the acoustic signal is not available. This introduces the need to build visual speech recognition systems for silent and whispered speech. However, almost all the recently proposed systems have been trained on…

Cited by 0 Source ↗Scholar ↗

2017

Audio-visual object localization and separation using low-rank and sparsity

Jie Pu, Yannis Panagakis, Stavros Petridis, Maja Pantic

ICASSP 2017accepted

The ability to localize visual objects that are associated with an audio source and at the same time seperate the audio signal is a corner stone in several audio-visual signal processing applications. Past efforts usually focused on localizing only the visual objects, without audio separation abilit…

Cited by 0 Source ↗Scholar ↗

2017

Deep Structured Learning for Facial Action Unit Intensity Estimation

Robert Walecki, Ognjen (Oggi) Rudovic, Vladimir Pavlovic, Bjoern Schuller, Maja Pantic

CVPR 2017poster

We consider the task of automated estimation of facial expression intensity. This involves estimation of multiple output variables (facial action units --- AUs) that are structurally dependent. Their structure arises from statistically induced co-occurrence patterns of AU intensity levels. Modeling…

Cited by 155 PDF ↗Scholar ↗

2017

DeepCoder: Semi-Parametric Variational Autoencoders for Automatic Facial Action Coding

Dieu Linh Tran, Robert Walecki, Ognjen (Oggi) Rudovic, Stefanos Eleftheriadis, Bjorn Schuller, Maja Pantic

ICCV 2017poster

Human face exhibits an inherent hierarchy in its representations (i.e., holistic facial expressions can be encoded via a set of facial action units (AUs) and their intensity). Variational (deep) auto-encoders (VAE) have shown great results in unsupervised extraction of hierarchical latent representa…

Cited by 56 PDF ↗Scholar ↗

2017

End-to-end visual speech recognition with LSTMS

Stavros Petridis, Zuwei Li, Maja Pantic

ICASSP 2017accepted

Traditional visual speech recognition systems consist of two stages, feature extraction and classification. Recently, several deep learning approaches have been presented which automatically extract features from the mouth images and aim to replace the feature extraction stage. However, research on…

Cited by 0 Source ↗Scholar ↗

2016

Copula Ordinal Regression for Joint Estimation of Facial Action Unit Intensity

Robert Walecki, Ognjen Rudovic, Vladimir Pavlovic, Maja Pantic

CVPR 2016poster

Joint modeling of the intensity of facial action units (AUs) from face images is challenging due to the large number of AUs (30+) and their intensity levels (6). This is in part due to the lack of suitable models that can efficiently handle such a large number of outputs/classes simultaneously, but…

Cited by 72 PDF ↗Scholar ↗

2016

Deep complementary bottleneck features for visual speech recognition

Stavros Petridis, Maja Pantic

ICASSP 2016accepted

Deep bottleneck features (DBNFs) have been used successfully in the past for acoustic speech recognition from audio. However, research on extracting DBNFs for visual speech recognition is very limited. In this work, we present an approach to extract deep bottleneck visual features based on deep auto…

Cited by 0 Source ↗Scholar ↗

2016

Joint Unsupervised Deformable Spatio-Temporal Alignment of Sequences

Lazaros Zafeiriou, Epameinondas Antonakos, Stefanos Zafeiriou, Maja Pantic

CVPR 2016poster

Typically, the problems of spatial and temporal alignment of sequences are considered disjoint. That is, in order to align two sequences, a methodology that (non)-rigidly aligns the images is first applied, followed by temporal alignment of the obtained aligned images. In this paper, we propose the…

Cited by 6 PDF ↗Scholar ↗

2015

Latent Trees for Estimating Intensity of Facial Action Units

Sebastian Kaltwang, Sinisa Todorovic, Maja Pantic

CVPR 2015poster

This paper is about estimating intensity levels of Facial Action Units (FAUs) in videos as an important and challenging step toward interpreting facial expressions. To address uncertainty in detections of facial landmark points, used as out input features, we formulate a new generative framework com…

Cited by 86 Source ↗PDF ↗Scholar ↗

2015

Multi-Conditional Latent Variable Model for Joint Facial Action Unit Detection

Stefanos Eleftheriadis, Ognjen Rudovic, Maja Pantic

ICCV 2015poster

We propose a novel multi-conditional latent variable model for simultaneous facial feature fusion and detection of facial action units. In our approach we exploit the structure-discovery capabilities of generative models such as Gaussian processes, and the discriminative power of classifiers such as…

Cited by 115 PDF ↗Scholar ↗

2015

Robust Statistical Face Frontalization

Christos Sagonas, Yannis Panagakis, Stefanos Zafeiriou, Maja Pantic

ICCV 2015poster

Recently, it has been shown that excellent results can be achieved in both facial landmark localization and pose-invariant face recognition. These breakthroughs are attributed to the efforts of the community to manually annotate facial images in many different poses and to collect 3D facial data. In…

Cited by 136 PDF ↗Scholar ↗