Objective: The goal of this study is to bridge the gap between laboratory assessment and real-world hearing experiences by integrating standard speech reception tests with video information, virtual reality, and assessment of listening effort. An integrated audio-video test bench is designed, allowing tests to be conducted in lifelike and ecologically valid conditions.

Methods: The setting up of the audio-video test bench is divided into three phases. The first phase involves the design and development of a multi-channel audio system for spatial audio reproduction, with a high capacity of recreating sound fields that closely approximate target sound environments – whether real or virtual with high spatial accuracy. The system, installed in a large audiometric booth, consists of 41 loudspeakers arranged on an irregular array, enabling Higher-Order Ambisonics (HOA) playback. The second phase involves the development of a Unity-based software platform capable of simultaneously managing the playback of auditory stimuli, the control of visual scenarios—presented through a VR headset—and the collection of speech reception accuracy data. The third phase focuses on integrating into the software system the acquisition of behavioral and physiological measures related to listening effort, including verbal response time, pupillometry, skin conductance, and heart rate. Once the system has been fully developed and integrated, it will be evaluated in an initial clinical-audiological application. In this phase, five typical real-life audio-visual scenarios will be reproduced, each representing different communication situations and background noise conditions. Examples include one-to-one conversations at short distances, such as those taking place in a café or a public park. The scenarios are created using 360° video recordings captured in the selected locations, combined with multichannel audio recordings. The speech test material used is the ITA Matrix sentence test, consisting of five-word sentences presented in an adaptive mode. Participants, either with normal hearing or with mild hearing loss will be asked to repeat each sentence they hear, allowing for the assessment of speech reception performance under realistic, ecologically valid conditions.

Expected results: It is expected that the proposed system will enable more accurate and ecologically valid assessment of speech reception and listening effort compared to traditional laboratory tests. The integration of behavioural and physiological measures is anticipated to provide a deeper understanding of real-world communication challenges. The outcomes will support the clinical adoption of immersive, multimodal testing protocols for hearing evaluation.