EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models

Maureen de Seyssel; Antony D’Avirro; Adina Williams; Emmanuel Dupoux

EMNLP 2024main5 citations

EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models

Maureen de Seyssel, Antony D’Avirro, Adina Williams, Emmanuel Dupoux

Abstract

We introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis. We apply this to two tasks: speech resynthesis and speech-to-speech translation. In both cases, the benchmark evaluates the ability of the model to encode emphasis in the speech input and accurately reproduce it in the output, potentially across a change of speaker and language. As part of the evaluation pipeline, we introduce EmphaClass, a new model that classifies emphasis at the frame or word level.

BibTeX

@inproceedings{seyssel-etal-2024-emphassess,
    title = "{E}mph{A}ssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models",
    author = "de Seyssel, Maureen  and
      D{'}Avirro, Antony  and
      Williams, Adina  and
      Dupoux, Emmanuel",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.30/",
    doi = "10.18653/v1/2024.emnlp-main.30",
    pages = "495--507"
}