Audio-only Condition Research Articles

Investigating the impact of visual access on trainee interpreters’ simultaneous interpreting performance

ABSTRACT Visual access has long been considered crucial for interpreters, providing essential kinesic cues such as gaze and facial expressions that enhance understanding and emphasis. Despite its acknowledged importance, empirical studies examining the impact of visual access on interpreters’ performance remain limited. To date, conclusive evidence establishing a direct link between visual access and interpreters’ performance is lacking. This study addresses this gap by employing a data-driven approach to explore the influence of visual components in source speeches on interpreting performance. A total of 48 graduate interpreter trainees (first language/L1: Chinese, second language/L2: English) participated in English-to-Chinese simultaneous interpreting tasks under both video and audio-only conditions. Two raters (L1: Chinese, L2: English) assessed their performances using three criteria: fidelity, target language quality (language), and fluency of delivery (delivery). Data analysis revealed that, overall, participants exhibited superior performance under the video condition (M = 4.92, SD = 1.62) compared to the audio-only condition (M = 4.19, SD = 1.67). Specifically, their performance in both fidelity and delivery was notably higher under the video condition (fidelity: M = 5.12, SD = 1.59; delivery: M = 4.58, SD = 0.23) than under the audio condition (fidelity: M = 4.38, SD = 1.63; delivery: M = 4.00, SD = 0.20). However, no statistically significant difference was observed in language performance between the two conditions. The implications are discussed at the end.

Audiovisual perception of interrupted speech by nonnative listeners.

The purpose of the present study was to examine the influence of visual cues in audiovisual perception of interrupted speech by nonnative English listeners and to identify the role of working memory, long-term memory retrieval, and vocabulary knowledge in audiovisual perception by nonnative listeners. The participants included 31 Mandarin-speaking English learners between 19 and 41 years of age. The perceptual stimuli were noise-filled periodically interrupted AzBio and QuickSIN sentences with or without visual cues that showed a male speaker uttering the sentences. In addition to sentence recognition, the listeners completed a semantic fluency task, verbal (operation span) and visuospatial (symmetry span) working memory tasks, and two vocabulary knowledge tests (Vocabulary Level Test and Lexical Test for Advanced Learners of English). The results revealed significantly better speech recognition in the audio-visual condition than the audio-only condition, but the magnitude of visual benefit was substantially attenuated for sentences that had limited semantic context. The listeners' vocabulary size in English played a key role in the restoration of missing speech information and audiovisual integration in the perception of interrupted speech. Meanwhile, the listeners' verbal working memory capacity played an important role in audiovisual integration especially for the difficult stimuli with limited semantic context.

Visual-articulatory cues facilitate children with CIs to better perceive Mandarin tones in sentences

Children with cochlear implants (CIs) face challenges in tonal perception under noise. Nevertheless, our previous research demonstrated that seeing visual-articulatory cues (speakers’ facial/head movements) benefited these children to perceive isolated tones better, particularly in noisy environments, with those implanted earlier gaining more benefits. However, tones in daily speech typically occur in sentence contexts where visual cues are largely reduced compared to those in isolated contexts. It was thus unclear if visual benefits on tonal perception still hold in these challenging sentence contexts. Therefore, this study tested 64 children with CIs and 64 age-matched NH children. Target tones in sentence-medial position were presented in audio-only (AO) or audiovisual (AV) conditions, in quiet and noisy environments. Children selected the target tone using a picture-point task. The results showed that, while NH children did not show any perception difference between AO and AV conditions, children with CIs significantly improved their perceptual accuracy from AO to AV conditions. The degree of improvement was negatively correlated with their implantation ages. Therefore, children with CIs were able to use visual-articulatory cues to facilitate their tonal perception even in sentence contexts, and earlier auditory experience might be important in shaping this ability.

Communicating environment protection from plastic waste via VR: Effects of realism and spatial presence on risk perception

This study tested the utility of virtual reality technologies in building public awareness of an environmental issue—the growing pollution of plastic waste in oceans. We conducted an experiment to test whether viewing of VR video would produce two anticipated immersive experiences (e.g., perceived realism and spatial presence) in severely polluted oceans, and to further examine their effects on participants’ pro-environmental attitudes with regards to reducing plastic waste. Results showed that VR viewing led to higher perceived realism and spatial presence in comparisons with 2D video and audio-only conditions; perceived realism contributed significantly to pro-environmental attitude, whereas spatial presence did not. The higher the perceived realism, the stronger the pro-environmental attitude. Implications of the findings for using VR technologies for effective environmental communication are discussed.

Comparison of Auditory and Visual Short-Term Memory Capabilities using a Serious Game Application

A comprehensive serious game application has been designed and implemented to examine the capacity and effective- ness of short-term auditory and visual memory, otherwise known as working memory in human subjects. Participants engaged in an adaptation of the well-known paired association game that entails turning over cards and recalling their placement within a 2D matrix structure of various resolutions. Each trial introduced either visual icons (vision only condition) or auditory objects (audio-only condition). User performance was evaluated through a detailed statistical analysis focusing only on the highest 6x8 resolution condition in the application. Findings suggest that visual memory did not conclusively outperform auditory memory in the context of this game. However, within the scope of auditory stimuli, familiar iconic sounds, such as excerpts of speech and commonplace sounds, were recalled more effectively than unfamiliar, synthetic sounds like parametric waveforms. Furthermore, performance appeared to be influenced by demographic factors, with male and younger subjects yielding superior results.

The effects of temporal cues, point-light displays, and faces on speech identification and listening effort.

Among the most robust findings in speech research is that the presence of a talking face improves the intelligibility of spoken language. Talking faces supplement the auditory signal by providing fine phonetic cues based on the placement of the articulators, as well as temporal cues to when speech is occurring. In this study, we varied the amount of information contained in the visual signal, ranging from temporal information alone to a natural talking face. Participants were presented with spoken sentences in energetic or informational masking in four different visual conditions: audio-only, a modulating circle providing temporal cues to salient features of the speech, a digitally rendered point-light display showing lip movement, and a natural talking face. We assessed both sentence identification accuracy and self-reported listening effort. Audiovisual benefit for intelligibility was observed for the natural face in both informational and energetic masking, but the digitally rendered point-light display only provided benefit in energetic masking. Intelligibility for speech accompanied by the modulating circle did not differ from the audio-only conditions in either masker type. Thus, the temporal cues used here were insufficient to improve speech intelligibility in noise, but some types of digital point-light displays may contain enough phonetic detail to produce modest improvements in speech identification in noise.

Visual Facial Enhancements Can Significantly Improve Speech Perception in the Presence of Noise.

Human speech perception is generally optimal in quiet environments, however it becomes more difficult and error prone in the presence of noise, such as other humans speaking nearby or ambient noise. In such situations, human speech perception is improved by speech reading, i.e., watching the movements of a speaker's mouth and face, either consciously as done by people with hearing loss or subconsciously by other humans. While previous work focused largely on speech perception of two-dimensional videos of faces, there is a gap in the research field focusing on facial features as seen in head-mounted displays, including the impacts of display resolution, and the effectiveness of visually enhancing a virtual human face on speech perception in the presence of noise. In this paper, we present a comparative user study ( N=21) in which we investigated an audio-only condition compared to two levels of head-mounted display resolution (1832×1920 or 916×960 pixels per eye) and two levels of the native or visually enhanced appearance of a virtual human, the latter consisting of an up-scaled facial representation and simulated lipstick (lip coloring) added to increase contrast. To understand effects on speech perception in noise, we measured participants' speech reception thresholds (SRTs) for each audio-visual stimulus condition. These thresholds indicate the decibel levels of the speech signal that are necessary for a listener to receive the speech correctly 50% of the time. First, we show that the display resolution significantly affected participants' ability to perceive the speech signal in noise, which has practical implications for the field, especially in social virtual environments. Second, we show that our visual enhancement method was able to compensate for limited display resolution and was generally preferred by participants. Specifically, our participants indicated that they benefited from the head scaling more than the added facial contrast from the simulated lipstick. We discuss relationships, implications, and guidelines for applications that aim to leverage such enhancements.

The effect of unisensory and multisensory information on lexical decision and free recall in young and older adults

Studies using simple low-level stimuli show that multisensory stimuli lead to greater improvements in processing speed for older adults than young adults. However, there is insufficient evidence to explain how these benefits influence performance for more complex processes such as judgement and memory tasks. This study examined how presenting stimuli in multiple sensory modalities (audio–visual) instead of one (audio-only or visual-only) may help older adults to improve their memory and cognitive processing compared to young adults. Young and older adults completed lexical decision (real word vs. pseudoword judgement) and word recall tasks, either independently, or in combination (dual-task), with and without perceptual noise. Older adults were better able to remember words when encoding independently. In contrast, young adults were better able to remember words when encoding in combination with lexical decisions. Both young and older adults had better word recall in the audio–visual condition compared with the audio-only condition. The findings indicate significant age differences when dealing with multiple tasks during encoding. Crucially, there is no greater multisensory benefit for older adults compared to young adults in more complex processes, rather multisensory stimuli can be useful in enhancing cognitive performance for both young and older adults.

Speech-derived haptic stimulation enhances speech recognition in a multi-talker background

Speech understanding, while effortless in quiet conditions, is challenging in noisy environments. Previous studies have revealed that a feasible approach to supplement speech-in-noise (SiN) perception consists in presenting speech-derived signals as haptic input. In the current study, we investigated whether the presentation of a vibrotactile signal derived from the speech temporal envelope can improve SiN intelligibility in a multi-talker background for untrained, normal-hearing listeners. We also determined if vibrotactile sensitivity, evaluated using vibrotactile detection thresholds, modulates the extent of audio-tactile SiN improvement. In practice, we measured participants’ speech recognition in a multi-talker noise without (audio-only) and with (audio-tactile) concurrent vibrotactile stimulation delivered in three schemes: to the left or right palm, or to both. Averaged across the three stimulation delivery schemes, the vibrotactile stimulation led to a significant improvement of 0.41 dB in SiN recognition when compared to the audio-only condition. Notably, there were no significant differences observed between the improvements in these delivery schemes. In addition, audio-tactile SiN benefit was significantly predicted by participants’ vibrotactile threshold levels and unimodal (audio-only) SiN performance. The extent of the improvement afforded by speech-envelope-derived vibrotactile stimulation was in line with previously uncovered vibrotactile enhancements of SiN perception in untrained listeners with no known hearing impairment. Overall, these results highlight the potential of concurrent vibrotactile stimulation to improve SiN recognition, especially in individuals with poor SiN perception abilities, and tentatively more so with increasing tactile sensitivity. Moreover, they lend support to the multimodal accounts of speech perception and research on tactile speech aid devices.

Aesthetic and physiological effects of naturalistic multimodal music listening

Compared to audio only (AO) conditions, audiovisual (AV) information can enhance the aesthetic experience of a music performance. However, such beneficial multimodal effects have yet to be studied in naturalistic music performance settings. Further, peripheral physiological correlates of aesthetic experiences are not well-understood. Here, participants were invited to a concert hall for piano performances of Bach, Messiaen, and Beethoven, which were presented in two conditions: AV and AO. They rated their aesthetic experience (AE) after each piece (Experiment 1 and 2), while peripheral signals (cardiorespiratory measures, skin conductance, and facial muscle activity) were continuously measured (Experiment 2). Factor scores of AE were significantly higher in the AV condition in both experiments. LF/HF ratio, a heart rhythm that represents activation of the sympathetic nervous system, was higher in the AO condition, suggesting increased arousal, likely caused by less predictable sound onsets in the AO condition. We present partial evidence that breathing was faster and facial muscle activity was higher in the AV condition, suggesting that observing a performer's movements likely enhances motor mimicry in these more voluntary peripheral measures. Further, zygomaticus (‘smiling’) muscle activity was a significant predictor of AE. Thus, we suggest physiological measures are related to AE, but at different levels: the more involuntary measures (i.e., heart rhythms) may reflect more sensory aspects, while the more voluntary measures (i.e., muscular control of breathing and facial responses) may reflect the liking aspect of an AE. In summary, we replicate and extend previous findings that AV information enhances AE in a naturalistic music performance setting. We further show that a combination of self-report and peripheral measures benefit a meaningful assessment of AE in naturalistic music performance settings.

A Visual Speech Intelligibility Benefit Based on Speech Rhythm

This study examined whether visual speech provides speech-rhythm information that perceivers can use in speech perception. This was tested by using speech that naturally varied in the familiarity of its rhythm. Thirty Australian English L1 listeners performed a speech perception in noise task with English sentences produced by three speakers: an English L1 speaker (familiar rhythm); an experienced English L2 speaker who had a weak foreign accent (familiar rhythm), and an inexperienced English L2 speaker who had a strong foreign accent (unfamiliar speech rhythm). The spoken sentences were presented in three conditions: Audio-Only (AO), Audio-Visual with mouth covered (AVm), and Audio-Visual (AV). Speech was best recognized in the AV condition regardless of the degree of foreign accent. However, speech recognition in AVm was better than AO for the speech with no foreign accent and with a weak accent, but not for the speech with a strong accent. A follow-up experiment was conducted that only used the speech with a strong foreign accent, under more audible conditions. The results also showed no difference between the AVm and AO conditions, indicating the null effect was not due to a floor effect. We propose that speech rhythm is conveyed by the motion of the jaw opening and closing, and perceivers use this information to better perceive speech in noise.

Differential audiovisual information processing in emotion recognition: An eye-tracking study.

Recent research has suggested that dynamic emotion recognition involves strong audiovisual association; that is, facial or vocal information alone automatically induces perceptual processes in the other modality. We hypothesized that different emotions may differ in the automaticity of audiovisual association, resulting in differential audiovisual information processing. Participants judged the emotion of a talking-head video under audiovisual, video-only (with no sound), and audio-only (with a static neutral face) conditions. Among the six basic emotions, disgust had the largest audiovisual advantage over the unimodal conditions in recognition accuracy. In addition, in the recognition of all the emotions except for disgust, participants' eye-movement patterns did not change significantly across the three conditions, suggesting mandatory audiovisual information processing. In contrast, in disgust recognition, participants' eye movements in the audiovisual condition were less eyes-focused than the video-only condition and more eyes-focused than the audio-only condition, suggesting that audio information in the audiovisual condition interfered with eye-movement planning for important features (eyes) for disgust. In addition, those whose eye-movement pattern was affected less by concurrent disgusted voice information benefited more in recognition accuracy. Disgust recognition is learned later in life and thus may involve a reduced amount of audiovisual associative learning. Consequently, audiovisual association in disgust recognition is less automatic and demands more attentional resources than other emotions. Thus, audiovisual information processing in emotion recognition depends on the automaticity of audiovisual association of the emotion resulting from associative learning. This finding has important implications for real-life emotion recognition and multimodal learning. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Fluency, comprehensibility, and accentedness in L2 speech: Examining the role of visual and acoustic information in listener judgments

The purpose of the present study was to examine (a) the effects of audio-visual information and (b) the role of temporal measurements associated with the perceptual ratings of fluency, accentedness, comprehensibility, pronunciation, and oral proficiency in second language (L2) speech samples of International Teaching Assistants (ITAs). American undergraduate students randomly assigned to audiovisual and audio-only (without seeing the speaker) conditions rated the two-minute speech samples of five ITAs on a seven-point scale. For acoustic analysis, several temporal measurements were used that were hypothesized to have an association with the ratings. Findings revealed that listeners without visual information judged L2 speech as less fluent and more accented although no significant difference in comprehensibility was found. L2 speakers were also judged to have better pronunciation and higher oral proficiency when the recordings were presented in video mode. The findings further indicated that among the acoustic measures examined, articulation rate was found to be associated with perceptual ratings. Implications for ITA training and language testing and assessment are discussed.

Do Visual Cues Aid Comprehension of a Dialogue?

Do Visual Cues Aid Comprehension of a Dialogue?

Role of sounds in perception of enclosure in urban street canyons

This study aims to examine the influence of sound on the perceptions of enclosures in urban street canyons with varying height-to-width ratios (H/Ws). Two typical urban streets, one narrow and one wide, were modelled with street widths of 6 m and 27 m, respectively. The H/W was varied from 0.5 to 6 in seven steps by changing the heights of the buildings on both sides of the street. Three-dimensional visual images were created in a virtual reality (VR) environment, and a synthesised car pass-by sound was convolved with impulse responses extracted from acoustic computer simulations as the sound stimuli. Laboratory experiments were conducted with a total of 41 participants. The experiments consisted of three sessions: 1) a visual-only condition, 2) a combined audio-visual condition, and 3) an audio-only condition. The subjective responses to the stimuli were rated in terms of the perceived enclosure, perceived spaciousness, perceived pleasantness, and perceived source width. The results from the experiment revealed that the perceived enclosure, perceived spaciousness, and perceived pleasantness of the visual stimuli decreased as the H/W increased. The presentation of sound significantly influenced the perceived enclosure and perceived spaciousness, whereas the perceived pleasantness was less affected by sound. The effects of sound on the subjective responses were greater for narrow streets than for wide streets.

Executive attention modulates the facilitating effect of electronic storybooks on information encoding in preschoolers

Interactive features and multimedia elements in electronic storybooks might enhance knowledge acquisition in children due to the playful learning experience they provide. However, to date, there is no systematic research on the long-term efficacy of storybooks, and the individual cognitive factors that influence information processing when using these apps. Therefore, in Experiment 1, we focused on long-term improvements. Children (M = 5.55 years, SD = 0.51, N = 33) were divided into an Interactive App group (N = 16) and a Print Book group (N = 17), then they were exposed to a story. Their recall performance was measured immediately after the exposure and three weeks later. In Experiment 2, we focused on individual differences in cognitive factors (working memory and sustained attention). Children (M = 5.56 years, SD = 0.62, N = 32) were exposed to three stories with interactive, multimedia-only elements and an audio-only condition. Caregivers were asked to fill out the ADHD Rating Scale-IV regarding each child. According to our results, in Experiment 1, children in the Interactive App group performed better compared to the Print Book group and this improvement persisted over time. In Experiment 2, we replicated the results of Experiment 1, however, children with poorer sustained attentional abilities performed worse in multimedia and interactive conditions compared to the audio-only condition. Our results indicate that electronic storybooks can facilitate learning because they enhance encoding efficacy. However, the benefit is only evident in children with good attentional control abilities. Our results guide parents and educators on how to choose and design age-appropriate applications for learning.

Speech Intelligibility in Reverberation is Reduced During Self-Rotation.

Speech intelligibility in cocktail party situations has been traditionally studied for stationary sound sources and stationary participants. Here, speech intelligibility and behavior were investigated during active self-rotation of standing participants in a spatialized speech test. We investigated if people would rotate to improve speech intelligibility, and we asked if knowing the target location would be further beneficial. Target sentences randomly appeared at one of four possible locations: 0°, ± 90°, 180° relative to the participant's initial orientation on each trial, while speech-shaped noise was presented from the front (0°). Participants responded naturally with self-rotating motion. Target sentences were presented either without (Audio-only) or with a picture of an avatar (Audio-Visual). In a baseline (Static) condition, people were standing still without visual location cues. Participants' self-orientation undershot the target location and orientations were close to acoustically optimal. Participants oriented more often in an acoustically optimal way, and speech intelligibility was higher in the Audio-Visual than in the Audio-only condition for the lateral targets. The intelligibility of the individual words in Audio-Visual and Audio-only increased during self-rotation towards the rear target, but it was reduced for the lateral targets when compared to Static, which could be mostly, but not fully, attributed to changes in spatial unmasking. Speech intelligibility prediction based on a model of static spatial unmasking considering self-rotations overestimated the participant performance by 1.4 dB. The results suggest that speech intelligibility is reduced during self-rotation, and that visual cues of location help to achieve more optimal self-rotations and better speech intelligibility.

I don't see what you're saying: The maluma/takete effect does not depend on the visual appearance of phonemes as they are articulated.

In contrast to the principle of arbitrariness, recent work has shown that language can iconically depict referents being talked about. One such example is the maluma/takete effect: an association between certain phonemes (e.g., those in maluma) and round shapes, and other phonemes (e.g., those in takete and spiky shapes). An open question has been whether this association is crossmodal (arising from phonemes' sound or kinesthetics) or unimodal (arising from phonemes' visual appearance). In the latter case, individuals may associate a person's rounded lips as they pronounce the /u/ in maluma with round shapes. We examined this hypothesis by having participants pair nonwords with shapes in either an audio-only condition (they only heard nonwords) or an audiovisual condition (they both heard nonwords and saw them articulated). We found no evidence that seeing nonwords articulated enhanced the maluma/takete effect. In fact, there was evidence that it decreased it in some cases. This was confirmed with a Bayesian analysis. These results eliminate a plausible explanation for the maluma/takete effect, as an instance of visual matching. We discuss the alternate possibility that it involves crossmodal associations.

Vehicle noise: comparison of loudness ratings in the field and the laboratory

Objective Distorted loudness perception is one of the main complaints of hearing aid users. Measuring loudness perception in the clinic as experienced in everyday listening situations is important for loudness-based hearing aid fitting. Little research has been done comparing loudness perception in the field and in the laboratory. Design Participants rated the loudness in the field and in the laboratory of 36 driving actions. The field measurements were recorded with a 360° camera and a tetrahedral microphone. The recorded stimuli, which are openly accessible, were presented in three conditions in the laboratory: 360° video recordings with a head-mounted display, video recordings with a desktop monitor and audio-only. Study samples Thirteen normal-hearing participants and 18 hearing-impaired participants with hearing aids. Results The driving actions were rated as louder in the laboratory than in the field for the condition with a desktop monitor and for the audio-only condition. The less realistic a laboratory condition was, the more likely it was for a participant to rate a driving action as louder. The field–laboratory loudness differences were bigger for louder sounds. Conclusion The results of this experiment indicate the importance of increasing realism and immersion when measuring loudness in the clinic.

Neural responses to naturalistic audiovisual speech are related to listening demand in cochlear implant users

There is a weak relationship between clinical and self-reported speech perception outcomes in cochlear implant (CI) listeners. Such poor correspondence may be due to differences in clinical and "real-world" listening environments and stimuli. Speech in the real world is often accompanied by visual cues, background environmental noise, and is generally in a conversational context, all factors that could affect listening demand. Thus, our objectives were to determine if brain responses to naturalistic speech could index speech perception and listening demand in CI users. Accordingly, we recorded high-density electroencephalogram (EEG) while CI users listened/watched a naturalistic stimulus (i.e., the television show, "The Office"). We used continuous EEG to quantify "speech neural tracking" (i.e., TRFs, temporal response functions) to the show's soundtrack and 8-12 Hz (alpha) brain rhythms commonly related to listening effort. Background noise at three different signal-to-noise ratios (SNRs), +5, +10, and +15 dB were presented to vary the difficulty of following the television show, mimicking a natural noisy environment. The task also included an audio-only (no video) condition. After each condition, participants subjectively rated listening demand and the degree of words and conversations they felt they understood. Fifteen CI users reported progressively higher degrees of listening demand and less words and conversation with increasing background noise. Listening demand and conversation understanding in the audio-only condition was comparable to that of the highest noise condition (+5 dB). Increasing background noise affected speech neural tracking at a group level, in addition to eliciting strong individual differences. Mixed effect modeling showed that listening demand and conversation understanding were correlated to early cortical speech tracking, such that high demand and low conversation understanding occurred with lower amplitude TRFs. In the high noise condition, greater listening demand was negatively correlated to parietal alpha power, where higher demand was related to lower alpha power. No significant correlations were observed between TRF/alpha and clinical speech perception scores. These results are similar to previous findings showing little relationship between clinical speech perception and quality-of-life in CI users. However, physiological responses to complex natural speech may provide an objective measure of aspects of quality-of-life measures like self-perceived listening demand.