18 accepted papers
Audio comprehension—including speech, non-speech sounds, and music—is essential for achieving human-level intelligence. Consequently, AI agents must demonstrate holistic audio understanding to qualify as generally intelligent. However, evaluating auditory intelligence comprehensively remains challen
We present Task 5 of the DCASE 2025 Challenge: an Audio Question Answering (AQA) benchmark spanning multiple domains of sound understanding. This task defines three QA subsets (Bioacoustics, Temporal Soundscapes, and Complex QA) to test audio-language models on interactive question-answering over di…
We introduce Music Flamingo, a novel large audio–language model, designed to advance music (including song) understanding in foundational audio models. While audio–language research has progressed rapidly, music remains challenging due to its dynamic, layered, and information-dense nature. Progress…
We present Audio Flamingo 3 (AF3), a fully open state-of-the-art (SOTA) large audio-language model that advances reasoning and understanding across speech, sound, and music. AF3 introduces: (i) AF-Whisper, a unified audio encoder trained using a novel strategy for joint representation learning acros…
Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in complex multimodal tasks. While MLLMs excel at visual perception and reasoning in third-person and egocentric videos, they are prone to hallucinations, generating coherent yet inaccurate responses. We present EGOILL
The MPEG-H standard for spatial audio proposes the rendering of multiple auditory objects (up to 16) and ambisonics to create a spatial audio scene. Convolution of these (and their early environmental reflections) with user-specific Head Related Impulse Responses (HRIRs), and a treatment of the late…
The ability to comprehend audio—which includes speech, non-speech sounds, and music—is crucial for AI agents to interact effectively with the world. We present MMAU, a novel benchmark designed to evaluate multimodal audio understanding models on tasks requiring expert-level knowledge and complex rea…
Speech enhancement (SE) is the fundamental task of enhancing the clarity and quality of speech in the presence of non-stationary additive noise. While deterministic deep learning models have been commonly employed for SE, recent research indicates that generative models, such as denoising diffusion…
Open-vocabulary audio-language models, like CLAP [1], offer a promising approach for zero-shot audio classification (ZSAC) by enabling classification with any arbitrary set of categories specified with natural language prompts. In this paper, we propose a simple but effective method to improve ZSAC…
Instruction Tuning (IT), the process of training large language models (LLMs) using instruction-response pairs, has emerged as the predominant method for transforming base pre-trained LLMs into open-domain conversational agents. While IT has achieved notable success and widespread adoption, its limi…
A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved performance in many downstream applications, including zero-shot a…
Perceiving and understanding non-speech sounds and non-verbal speech is essential to making decisions that help us interact with our surroundings. In this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model (LALM) with Advanced Audio Understanding and Complex Reasoning Abiliti…
We present RECAP (REtrieval-Augmented Audio CAPtioning), a novel and effective audio captioning system that generates captions conditioned on an input audio and other captions similar to the audio retrieved from a datastore. Additionally, our proposed method can transfer to any domain without the ne…
A novel version of Békésy audiometry is developed that is suitable for remote administration via an app, and for DSP integration into headphones. To demonstrate the efficacy and usefulness of the approach, experiments were performed with 32 participants with a range of ages and hearing losses. In ex…
Incorporating individualized head-related transfer functions (HRTFs) into a high fidelity sound engine can further improve the perceived quality and realism of binaurally-rendered spatial audio. Traditional methods to measure individual HRTFs tend to be cumbersome, expensive and require physical acc…
We introduce a novel algorithm for the decomposition of a broadband soundfield into its component plane-waves. The algorithm, termed Sequential Direction Detection, decomposes the sound-field into L plane waves by recursively minimizing an objective function that determines the plane-wave directions…
The nonuniform fast Fourier transform comprises a set of algorithms which approximately interpolate the usual discrete Fourier transform in the time and/or frequency domain. A nonuniform fast Fourier transform based on the fast multipole method was previously developed but passed over in favor of ot…
Any acoustic sensor disturbs the spatial acoustic field to certain extent, and a recorded field is different from a field that would have existed if a sensor were absent. Recovery of the original (incident) field is a fundamental task in spatial audio. For some sensor geometries, the disturbance of…