Sabato Marco Siniscalchi

29 accepted papers

2026

ALIGNING GENERATIVE SPEECH ENHANCEMENT WITH PERCEPTUAL FEEDBACK

Haoyang Li, Nana Hou, Yuchen Hu, Jixun Yao, Sabato Marco Siniscalchi, Norwegian University of Science and Technology +4

ICASSP 2026oral

Language Model (LM)-based speech enhancement (SE) has recently emerged as a promising direction, but existing approaches predominantly rely on token-level likelihood objectives that weakly reflect human perception. This mismatch limits progress, as optimizing signal accuracy does not always improve…

Cited by 0 Source ↗PDF ↗Scholar ↗

2026

FEW-SHOT AND PSEUDO-LABEL GUIDED SPEECH QUALITY EVALUATION WITH LARGE LANGUAGE MODELS

Ryandhimas E. Zezario, Szu-Wei Fu, Sabato Marco Siniscalchi, Hsin-Min Wang

ICASSP 2026oral

In this paper, we introduce GatherMOS, a novel framework that leverages large language models (LLM) as meta-evaluators to aggregate diverse signals into quality predictions. GatherMOS integrates lightweight acoustic descriptors with pseudo-labels from DNSMOS and VQScore, enabling the LLM to reason o…

Cited by 0 Source ↗PDF ↗Scholar ↗

2026

TimeOmni-1: Incentivizing Complex Reasoning with Time Series in Large Language Models

Tong Guan, Zijie Meng, Dianqi Li, Shiyu Wang, Chao-Han Huck Yang, Qingsong Wen +4

ICLR 2026poster

Recent advances in multimodal time series learning underscore a paradigm shift from analytics centered on basic patterns toward advanced time series understanding and reasoning. However, existing multimodal time series datasets mostly remain at the level of surface alignment and question answering,…

Cited by 0 Source ↗code ↗Scholar ↗

2025

A Study on Zero-shot Non-intrusive Speech Assessment using Large Language Models

Ryandhimas E. Zezario, Sabato Marco Siniscalchi, Hsin-Min Wang, Yu Tsao

ICASSP 2025accepted

This work investigates two strategies for zero-shot non-intrusive speech assessment leveraging large language models. First, we explore the audio analysis capabilities of GPT-4o. Second, we propose GPT-Whisper, which uses Whisper as an audio-to-text module and evaluates the text’s naturalness via ta…

Cited by 0 Source ↗Scholar ↗

2025

An Explicit Consistency-Preserving Loss Function for Phase Reconstruction and Speech Enhancement

Pin-Jui Ku, Chun-Wei Ho, Hao Yen, Sabato Marco Siniscalchi, Chin-Hui Lee

ICASSP 2025accepted

In this work, we propose a novel consistency-preserving loss function for recovering the phase information in the context of phase reconstruction (PR) and speech enhancement (SE). Different from conventional techniques that directly estimate the phase using a deep model, our idea is to exploit ad-ho…

Cited by 0 Source ↗Scholar ↗

2025

Bilingual Dual-Head Deep Model for Parkinson's Disease Detection from Speech

Moreno La Quatra, Juan Rafael Orozco-Arroyave, Sabato Marco Siniscalchi

ICASSP 2025accepted

This work aims to tackle the Parkinson’s disease (PD) detection problem from the speech signal in a bilingual setting by proposing an ad-hoc dual-head deep neural architecture for type-based binary classification. One head is specialized for diadochokinetic patterns. The other head looks for natural…

Cited by 0 Source ↗Scholar ↗

2025

MISP-Meeting: A Real-World Dataset with Multimodal Cues for Long-form Meeting Transcription and Summarization

HangChen HangChen, Chao-Han Huck Yang, Jia-Chen Gu, Sabato Marco Siniscalchi, Jun Du

ACL 2025long

We introduce MISP-Meeting, a new real-world, multimodal dataset that covers subject-oriented long-form content. MISP-Meeting integrates information from speech, vision, and text modalities to facilitate automatic meeting transcription and summarization (AMTS). Challenging conditions in human meeting…

Cited by 0 Source ↗PDF ↗code ↗Scholar ↗

2025

MSEMG: Surface Electromyography Denoising with a Mamba-based Efficient Network

Yu-Tung Liu, Kuan-Chen Wang, Rong Chao, Sabato Marco Siniscalchi, Ping-Cheng Yeh, Yu Tsao

ICASSP 2025accepted

Surface electromyography (sEMG) recordings can be contaminated by electrocardiogram (ECG) signals when the monitored muscle is closed to the heart. Traditional signal processing-based approaches, such as high-pass filtering and template subtraction, have been used to remove ECG interference but are…

Cited by 0 Source ↗Scholar ↗

2025

voc2vec: A Foundation Model for Non-Verbal Vocalization

Alkis Koudounas, Moreno La Quatra, Sabato Marco Siniscalchi, Elena Baralis

ICASSP 2025accepted

Speech foundation models have demonstrated exceptional capabilities in speech-related tasks. Nevertheless, these models often struggle with non-verbal audio data, such as vocalizations, baby crying, etc., which are critical for various real-world applications. Audio foundation models well handle non…

Cited by 0 Source ↗Scholar ↗

2024

Boosting End-to-End Multilingual Phoneme Recognition Through Exploiting Universal Speech Attributes Constraints

Hao Yen, Sabato Marco Siniscalchi, Chin-Hui Lee

ICASSP 2024accepted

We propose a first step toward multilingual end-to-end automatic speech recognition (ASR) by integrating knowledge about speech articulators. The key idea is to leverage a rich set of fundamental units that can be defined "universally" across all spoken languages, referred to as speech attributes, n…

Cited by 0 Source ↗Scholar ↗

2024

It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

CHEN CHEN, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi, Pin-Yu Chen, EngSiong Chng +1

ICLR 2024poster

Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output. Specifically, an LLM is utilized to carry out a direct mapping from the N-best hypotheses list generated by…

Cited by 27 Source ↗PDF ↗Scholar ↗

2024

Speech Analysis of Language Varieties in Italy

Moreno La Quatra, Alkis Koudounas, Elena Baralis, Sabato Marco Siniscalchi

COLING 2024main

Italy exhibits rich linguistic diversity across its territory due to the distinct regional languages spoken in different areas. Recent advances in self-supervised learning provide new opportunities to analyze Italy’s linguistic varieties using speech data alone. This includes the potential to levera…

Cited by 3 Source ↗PDF ↗code ↗Scholar ↗

2024

The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction

Shilong Wu, Chenxi Wang, Hang Chen, Yusheng Dai, Chenyue Zhang, Ruoyu Wang +9

ICASSP 2024accepted

Previous Multimodal Information based Speech Processing (MISP) challenges mainly focused on audio-visual speech recognition (AVSR) with commendable success. However, the most advanced back-end recognition systems often hit performance limits due to the complex acoustic environments. This has prompte…

Cited by 0 Source ↗Scholar ↗

2023

A Quantum Kernel Learning Approach to Acoustic Modeling for Spoken Command Recognition

Chao-Han Huck Yang, Bo Li, Yu Zhang, Nanxin Chen, Tara N. Sainath, Sabato Marco Siniscalchi +1

ICASSP 2023accepted

We propose a quantum kernel learning (QKL) framework to address the inherent data sparsity issues often encountered in training large-scare acoustic models in low-resource scenarios. We project acoustic features based on classical-to-quantum feature encoding. Different from existing quantum convolut…

Cited by 11 Source ↗Scholar ↗

2023

HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

CHEN CHEN, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, EngSiong Chng

NeurIPS 2023poster

Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems to attain human parity on several publicly available clean speech datasets. However, even state-of-the-art ASR systems experience performance degradation when confronted with adverse conditions, as a well-tr…

Cited by 59 Source ↗PDF ↗code ↗Scholar ↗

2023

Summary on the Multimodal Information Based Speech Processing (MISP) 2022 Challenge

Hang Chen, Shilong Wu, Yusheng Dai, Zhe Wang, Jun Du, Chin-Hui Lee +9

ICASSP 2023accepted

The Multimodal Information based Speech Processing (MISP) 2022 challenge aimed to enhance speech processing performance in harsh acoustic environments by leveraging additional modalities such as video or text. The challenge included two tracks: audio-visual speaker diarization (AVSD) and audio-visua…

Cited by 0 Source ↗Scholar ↗

2023

The Multimodal Information Based Speech Processing (Misp) 2022 Challenge: Audio-Visual Diarization And Recognition

Zhe Wang, Shilong Wu, Hang Chen, Mao-Kui He, Jun Du, Chin-Hui Lee +9

ICASSP 2023accepted

The Multi-modal Information based Speech Processing (MISP) challenge aims to extend the application of signal processing technology in specific scenarios by promoting the research into wake-up words, speaker diarization, speech recognition, and other technologies. The MISP2022 challenge has two trac…

Cited by 0 Source ↗Scholar ↗

2022

A Variational Bayesian Approach to Learning Latent Variables for Acoustic Knowledge Transfer

Hu Hu, Sabato Marco Siniscalchi, Chao-Han Huck Yang, Chin-Hui Lee

ICASSP 2022accepted

We propose a variational Bayesian (VB) approach to learning distributions of latent variables in deep neural network (DNN) models for cross-domain knowledge transfer, to address acoustic mismatches between training and testing conditions. Instead of carrying out point estimation in conventional maxi…

Cited by 0 Source ↗Scholar ↗

2022

The First Multimodal Information Based Speech Processing (Misp) Challenge: Data, Tasks, Baselines And Results

Hang Chen, Hengshun Zhou, Jun Du, Chin-Hui Lee, Jingdong Chen, Shinji Watanabe +7

ICASSP 2022accepted

In this paper we discuss the rational of the Multi-model Information based Speech Processing (MISP) Challenge, and provide a detailed description of the data recorded, the two evaluation tasks and the corresponding baselines, followed by a summary of submitted systems and evaluation results. The MIS…

Cited by 0 Source ↗Scholar ↗

2021

A Two-Stage Approach to Device-Robust Acoustic Scene Classification

Hu Hu, Chao-Han Huck Yang, Xianjun Xia, Xue Bai, Xin Tang, Yajian Wang +10

ICASSP 2021accepted

To improve device robustness, a highly desirable key feature of a competitive data-driven acoustic scene classification (ASC) system, a novel two-stage system based on fully convolutional neural networks (CNNs) is proposed. Our two-stage system leverages on an ad-hoc score combination based on two C…

Cited by 0 Source ↗Scholar ↗

2021

A Two-Stage Deep Modeling Approach to Articulatory Inversion

Abdolreza Sabzi Shahrebabaki, Negar Olfati, Ali Shariq Imran, Magne Hallstein Johnsen, Sabato Marco Siniscalchi, Torbjørn Svendsen

ICASSP 2021accepted

This paper proposes a two-stage deep feed-forward neural network (DNN) to tackle the acoustic-to-articulatory inversion (AAI) problem. DNNs are a viable solution for the AAI task, but the temporal continuity of the estimated articulatory values has not been exploited properly when a DNN is employed.…

Cited by 0 Source ↗Scholar ↗

2021

Decentralizing Feature Extraction with Quantum Convolutional Neural Network for Automatic Speech Recognition

Chao-Han Huck Yang, Jun Qi, Samuel Yen-Chi Chen, Pin-Yu Chen, Sabato Marco Siniscalchi, Xiaoli Ma +1

ICASSP 2021accepted

We propose a novel decentralized feature extraction approach in federated learning to address privacy-preservation issues for speech recognition. It is built upon a quantum convolutional neural network (QCNN) composed of a quantum circuit encoder for feature extraction, and a recurrent neural networ…

Cited by 0 Source ↗Scholar ↗

2020

A Cross-Task Transfer Learning Approach to Adapting Deep Speech Enhancement Models to Unseen Background Noise Using Paired Senone Classifiers

Sicheng Wang, Wei Li, Sabato Marco Siniscalchi, Chin-Hui Lee

ICASSP 2020accepted

We propose an environment adaptation approach that improves deep speech enhancement models via minimizing the Kullback-Leibler divergence between posterior probabilities produced by a multi-condition senone classifier (teacher) fed with noisy speech features and a clean-condition senone classifier (…

Cited by 0 Source ↗Scholar ↗

2020

Tensor-To-Vector Regression for Multi-Channel Speech Enhancement Based on Tensor-Train Network

Jun Qi, Hu Hu, Yannan Wang, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Chin-Hui Lee

ICASSP 2020accepted

We propose a tensor-to-vector regression approach to multi-channel speech enhancement in order to address the issue of input size explosion and hidden-layer size expansion. The key idea is to cast the conventional deep neural network (DNN) based vector-to-vector regression formulation under a tensor…

Cited by 0 Source ↗Scholar ↗

2019

Exploring Retraining-free Speech Recognition for Intra-sentential Code-switching

Zhen Huang, Xiaodan Zhuang, Daben Liu, Xiaoqiang Xiao, Yuchen Zhang, Sabato Marco Siniscalchi

ICASSP 2019accepted

Code Switching refers to the phenomenon of changing languages within a sentence or discourse, and it represents a challenge for conventional automatic speech recognition systems deployed to tackle a single target language. The code switching problem is complicated by the lack of multi-lingual traini…

Cited by 0 Source ↗Scholar ↗

2019

Improving Audio-visual Speech Recognition Performance with Cross-modal Student-teacher Training

Wei Li, Sicheng Wang, Ming Lei, Sabato Marco Siniscalchi, Chin-Hui Lee

ICASSP 2019accepted

In this paper, we propose a cross-modal student-teacher learning framework to make a full use of externally abundant acoustic data in addition to a given task-specific audio-visual training database for improving speech recognition performance under the low signal-to-noise-ratio (SNR) and acoustic m…

Cited by 0 Source ↗Scholar ↗

2018

Improving Mandarin Tone Mispronunciation Detection for Non-Native Learners with Soft-Target Tone Labels and BLSTM-Based Deep Models

Wei Li, Nancy F. Chen, Sabato Marco Siniscalchi, Chin-Hui Lee

ICASSP 2018accepted

We propose three techniques to improve mispronunciation detection of Mandarin tones of second language (L2) learners using tone-based extended recognition network (ERN). First, we extend our model from deep neural network (DNN) to bidirectionallon-short-term memory (BLSTM) in order to model tone-lev…

Cited by 0 Source ↗Scholar ↗

2017

A transfer learning and progressive stacking approach to reducing deep model sizes with an application to speech enhancement

Sicheng Wang, Kehuang Li, Zhen Huang, Sabato Marco Siniscalchi, Chin-Hui Lee

ICASSP 2017accepted

Leveraging upon transfer learning, we distill the knowledge in a conventional wide and deep neural network (DNN) into a narrower yet deeper model with fewer parameters and comparable system performance for speech enhancement. We present three transfer-learning solutions to accomplish our goal. First…

Cited by 0 Source ↗Scholar ↗

2016

Improving non-native mispronunciation detection and enriching diagnostic feedback with DNN-based speech attribute modeling

Wei Li, Sabato Marco Siniscalchi, Nancy F. Chen, Chin-Hui Lee

ICASSP 2016accepted

We propose the use of speech attributes, such as voicing and aspiration, to address two key research issues in computer assisted pronunciation training (CAPT) for L2 learners, namely detecting mispronunciation and providing diagnostic feedback. To improve the performance we focus on mispronunciation…

Cited by 0 Source ↗Scholar ↗