Multiple Speakers Research Articles

Specifying Challenges in Transcribing Covert Recordings: Implications for Forensic Transcription

Covert audio recordings feature in the criminal justice system in a variety of guises, either on their own or accompanied by video. If legally obtained, such recordings can provide important forensic evidence. However, the quality of these potentially valuable evidential recordings is often very poor and their content indistinct, to the extent that a jury requires an accompanying transcript. At present, in many international jurisdictions, these transcriptions are produced by investigating police officers involved in the case, but transcription is a highly complex, meticulous and onerous task, and police officers are untrained and have a vested interest in the influence of the transcript on a case, which gives rise to potential inaccuracy. This paper reports the design and results of a controlled transcription experiment in which eight linguistically trained professional transcribers produced transcripts for an audio recording of a conversation between five adults in a busy restaurant. In the context of covert recordings, this recording shares many of the typical features of covert forensic recordings, including the presence of multiple speakers, background noise and use of non-specialist recording equipment. We present a detailed qualitative and quantitative comparison of the transcripts, identifying areas of agreement and disagreement in (a) speaker attribution and (b) the representation of the linguistic content. We find that disagreement between the transcriptions is frequent and various in nature; the most common causes are identified as (i) omission of speech that is included in other transcripts, (ii) variation in the representation of turns, (iii) orthographic variation seemingly motivated by phonetic similarity, and (iv) orthographic variation seemingly not motivated by phonetic similarity. We argue that the variable nature of the transcription of “challenging” audio recordings must be considered in forensic contexts and make recommendations for improving practice in the production of forensic transcriptions.

Attention Differentially Affects Acoustic and Phonetic Feature Encoding in a Multispeaker Environment.

Humans have the remarkable ability to selectively focus on a single talker in the midst of other competing talkers. The neural mechanisms that underlie this phenomenon remain incompletely understood. In particular, there has been longstanding debate over whether attention operates at an early or late stage in the speech processing hierarchy. One way to better understand this is to examine how attention might differentially affect neurophysiological indices of hierarchical acoustic and linguistic speech representations. In this study, we do this by using encoding models to identify neural correlates of speech processing at various levels of representation. Specifically, we recorded EEG from fourteen human subjects (nine female and five male) during a “cocktail party” attention experiment. Model comparisons based on these data revealed phonetic feature processing for attended, but not unattended speech. Furthermore, we show that attention specifically enhances isolated indices of phonetic feature processing, but that such attention effects are not apparent for isolated measures of acoustic processing. These results provide new insights into the effects of attention on different prelexical representations of speech, insights that complement recent anatomic accounts of the hierarchical encoding of attended speech. Furthermore, our findings support the notion that, for attended speech, phonetic features are processed as a distinct stage, separate from the processing of the speech acoustics.SIGNIFICANCE STATEMENT Humans are very good at paying attention to one speaker in an environment with multiple speakers. However, the details of how attended and unattended speech are processed differently by the brain is not completely clear. Here, we explore how attention affects the processing of the acoustic sounds of speech as well as the mapping of those sounds onto categorical phonetic features. We find evidence of categorical phonetic feature processing for attended, but not unattended speech. Furthermore, we find evidence that categorical phonetic feature processing is enhanced by attention, but acoustic processing is not. These findings add an important new layer in our understanding of how the human brain solves the cocktail party problem.

Design Proposal for Sign Language Services in TV Broadcasting from the Perspective of People Who Are Deaf or Hard of Hearing

Sign language services are provided so that people with hearing loss are not alienated from socially and politically important information through TV broadcasting. In this paper, we conducted a user survey and evaluation of the current sign language services for deaf or hard-of-hearing (DHH) people, and solutions were proposed for the problems found in the course of the analyses. To this end, a total of five stages of research were conducted. First, the communication problems experienced by DHH individuals and previous studies on their language and information acquisition were investigated. Second, the most typical types of information delivery channels via TV were defined as news, discussions, and weather reports, and by investigating the actual sign language service cases for each type, three visual information delivery elements were identified: sign language interpreters, reference videos, and subtitles. Third, a preference survey, an interview survey, and an eye tracker experiment on the DHH participants were conducted with varying arrangement options of information delivery elements. Fourth, based on the results of the investigations and experiments, the options to be considered when arranging information delivery elements were compiled. The results showed that the sign language interpreter, which is the first element of information delivery, should be presented in a size clearly visible because the visibility of their facial expressions is important. In addition, it is recommended to present the interpreter without a background since DHH participants did not prefer the presence of a background. As for subtitles, which is the third element of information delivery, it was confirmed that the provision of sign language interpretation and subtitles together helped DHH participants to understand the contents more quickly and accurately. Moreover, if there are multiple speakers, individual subtitles for each speaker should be provided so that the viewers can understand who is talking. Reference videos, which are mainly placed on the screen background, the second information delivery element, were considered less important to DHH participants compared to sign language interpreters and subtitles, and it was found that DHH participants preferred reference videos to be visually separated from sign language interpreters. Fifth, based on the overall results of the study, a screen layout design was proposed for each type of information delivery element for DHH people. Contrary to the general conception that there would be no problem in viewing information-delivering TV broadcasts by DHH people simply by placing a sign language interpreter on the screen, the results of this study confirmed that a more delicate screen layout design is necessary for DHH people. It is expected that this study will serve as a helpful guide in providing better sign language services for TV broadcasts that can be conveniently viewed by both DHH and non-disabled people.

Physics-based sound radiation estimation from multiple speakers by combined lumped parameter and reduced-order finite element modeling

In this paper a virtual sensor is derived that can perform full field estimation of the sound field radiating from a number of loudspeakers in the time domain. The virtual sensor scheme contains two parts: Firstly, a time-domain physics-based numerical system level model that includes both the electro-mechanical behavior of the loudspeaker and the resulting wave propagation. Secondly, a Kalman filter in which the numerical model and a limited set of microphone measurements are combined to perform full field estimation. The loudspeaker dynamics are captured in a lumped parameter model and the resulting wave propagation is captured in a finite element model. To reduce the calculational complexity of the system level model, Krylov subspace based reduced order modeling is applied on the finite element model, leading to a significant reduction of the number of states. The system level model is brought to a discrete time state–space format, which allows for efficient implementation in the Kalman filter. The method is validated using two setups that use both vented and closed box loudspeakers, leading to improved estimation results as compared to pure simulation results, thus showing the merits of the virtual sensor. Furthermore, the importance of including the lumped parameter speaker model in the system level model is shown by comparing the estimation results with and without the inclusion of the lumped parameter model.

Child-directed and overheard input from different speakers in two distinct cultures.

Despite the fact that in most communities interaction occurs between the child and multiple speakers, most previous research on input to children focused on input from mothers. We annotated recordings of Sesotho-learning toddlers living in non-industrial Lesotho in South Africa, and French-learning toddlers living in urban regions in France. We examined who produced the input (mothers, other children, adults), how much input was child directed, and whether and how it varied across speakers. As expected, mothers contributed most of the input in the French recordings. However, in the Sesotho recordings, input from other children was more common than input from mothers or other adults. Child-directed speech from all speakers in both cultural groups showed similar qualitative modifications. Our findings suggest that input from other children is prevalent and has similar features as child-directed from adults described in previous work, inviting cross-cultural research into the effects of input from other children.

Distributed Multiple Speaker Tracking Based on Unscented Particle Filter and Data Association in Microphone Array Networks

Distributed Multiple Speaker Tracking Based on Unscented Particle Filter and Data Association in Microphone Array Networks

Overlapped speech detection using phase features.

Simultaneous speech of multiple speakers is known as overlapped speech, which causes problems for speech recognition and speaker diarization systems. The present work uses previously less utilized signal phase information in the task of overlapped speech detection. In this context, Instantaneous Frequency Cosine Coefficient (IFCC) and Modified Group Delay Cepstral Coefficient (MGDCC) features are explored. IFCC captures the time-varying phase characteristics, while MGDCC represents the frequency-varying information of the phase spectrum. A Convolutional Neural Network and Long Short-Term Memory (CNN-LSTM)-based classifier is used for the classification. The present work uses synthetically generated overlapped speech from the GRID corpus. The proposed method is benchmarked against three baseline approaches that use magnitude spectrum features. It is observed that the combination of IFCC and MGDCC features with CNN-LSTM classifier provides better performance than the baselines. The combination of phase features with magnitude-based MFCC feature provides the best performance, indicating the importance of complementary information. The present study also investigates the effect of segment duration, genders, and number of simultaneous speakers on the overlapped speech detection system. Finally, the proposed method is also evaluated on real overlapped data from the AMI corpus.

An audio enhancement system to improve intelligibility for social-awareness in HRI

Improving the ability to interact through voice with a robot is still a challenge especially in real environments where multiple speakers coexist. This work has evaluated a proposal based on improving the intelligibility of the voice information that feeds an existing ASR service in the network and in conditions similar to those that could occur in a care centre for the elderly. The results indicate the feasibility and improvement of a proposal based on the use of an embedded microphone array and the use of a simple beamforming and masking technique. The system has been evaluated with 12 people and results obtained for time responsiveness indicate that the system would allow natural interaction with voice. It is shown to be necessary to incorporate a system to properly employ the masking algorithm, through the intelligent and stable estimation of the interfering signals. In addition, this approach allows to fix as sources of interest other speakers not located in the vicinity of the robot.

Effects of Sinusoidal Model on Non-Parallel Voice Conversion with Adversarial Learning

Voice conversion (VC) transforms the speaking style of a source speaker to the speaking style of a target speaker by keeping linguistic information unchanged. Traditional VC techniques rely on parallel recordings of multiple speakers uttering the same sentences. Earlier approaches mainly find a mapping between the given source–target speakers, which contain pairs of similar utterances spoken by different speakers. However, parallel data are computationally expensive and difficult to collect. Non-parallel VC remains an interesting but challenging speech processing task. To address this limitation, we propose a method that allows a non-parallel many-to-many voice conversion by using a generative adversarial network. To the best of the authors’ knowledge, our study is the first one that employs a sinusoidal model with continuous parameters to generate converted speech signals. Our method involves only several minutes of training examples without parallel utterances or time alignment procedures, where the source–target speakers are entirely unseen by the training dataset. Moreover, empirical study is carried out on the publicly available CSTR VCTK corpus. Our conclusions indicate that the proposed method reached the state-of-the-art results in speaker similarity to the utterance produced by the target speaker, while suggesting important structural ones to be further analyzed by experts.

Extracting the Auditory Attention in a Dual-Speaker Scenario From EEG Using a Joint CNN-LSTM Model.

Human brain performs remarkably well in segregating a particular speaker from interfering ones in a multispeaker scenario. We can quantitatively evaluate the segregation capability by modeling a relationship between the speech signals present in an auditory scene, and the listener's cortical signals measured using electroencephalography (EEG). This has opened up avenues to integrate neuro-feedback into hearing aids where the device can infer user's attention and enhance the attended speaker. Commonly used algorithms to infer the auditory attention are based on linear systems theory where cues such as speech envelopes are mapped on to the EEG signals. Here, we present a joint convolutional neural network (CNN)—long short-term memory (LSTM) model to infer the auditory attention. Our joint CNN-LSTM model takes the EEG signals and the spectrogram of the multiple speakers as inputs and classifies the attention to one of the speakers. We evaluated the reliability of our network using three different datasets comprising of 61 subjects, where each subject undertook a dual-speaker experiment. The three datasets analyzed corresponded to speech stimuli presented in three different languages namely German, Danish, and Dutch. Using the proposed joint CNN-LSTM model, we obtained a median decoding accuracy of 77.2% at a trial duration of 3 s. Furthermore, we evaluated the amount of sparsity that the model can tolerate by means of magnitude pruning and found a tolerance of up to 50% sparsity without substantial loss of decoding accuracy.

Deep Gaussian process based multi-speaker speech synthesis with latent speaker representation

This paper proposes deep Gaussian process (DGP)-based frameworks for multi-speaker speech synthesis and speaker representation learning. A DGP has a deep architecture of Bayesian kernel regression, and it has been reported that DGP-based single speaker speech synthesis outperforms deep neural network (DNN)-based ones in the framework of statistical parametric speech synthesis. By extending this method to multiple speakers, it is expected that higher speech quality can be achieved with a smaller number of training utterances from each speaker. To apply DGPs to multi-speaker speech synthesis, we propose two methods: one using DGP with one-hot speaker codes, and the other using a deep Gaussian process latent variable model (DGPLVM). The DGP with one-hot speaker codes uses additional GP layers to transform speaker codes into latent speaker representations. The DGPLVM directly models the distribution of latent speaker representations and learns it jointly with acoustic model parameters. In this method, acoustic speaker similarity is expressed in terms of the similarity of the speaker representations, and thus, the voices of similar speakers are efficiently modeled. We experimentally evaluated the performance of the proposed methods in comparison with those of conventional DNN and variational autoencoder (VAE)-based frameworks, in terms of acoustic feature distortion and subjective speech quality. The experimental results demonstrate that (1) the proposed DGP-based and DGPLVM-based methods improve subjective speech quality compared with a feed-forward DNN-based method, and (2) even when the amount of training data for target speakers is limited, the DGPLVM-based method outperforms other methods, including the VAE-based one. Additionally, (3) by using a speaker representation randomly sampled from the learned speaker space, the DGPLVM-based method can generate voices of non-existent speakers.

Studies on association characteristics between vowels and visual colors using multiple speakers' speech

Studies on association characteristics between vowels and visual colors using multiple speakers' speech

Adult perception of stop consonant voicing in American-English-learning toddlers: Voice onset time and secondary cues.

Most studies of speech perception employ highly controlled stimuli. It is not always clear how such results extend to the processing of natural speech. In a series of experiments, we progressively explored the role of voice onset time (VOT) and potential secondary cues in adult labeling of stressed syllable-initial /b d p t/ produced by typically developing two-year-old learners of American English. Taken together, the results show the following: (a) Adult listeners show phoneme boundaries in labeling functions comparable to what have been established for adult speech. (b) Adult listeners can be sensitive to distributional properties of the stimulus set, even in a study that employs highly varied naturalistic productions from multiple speakers. (c) Secondary cues are available in the speech of two-year-olds, and these may influence listener judgments. Cues may differ across places of articulation and the VOT continuum. These results can lend insight into how clinicians judge child speech during assessment and also have implications for our understanding of the role of primary and secondary acoustic cues in adult perception of child speech.

Short-Term Audiovisual Spatial Training Enhances Electrophysiological Correlates of Auditory Selective Spatial Attention.

Audiovisual cross-modal training has been proposed as a tool to improve human spatial hearing. Here, we investigated training-induced modulations of event-related potential (ERP) components that have been associated with processes of auditory selective spatial attention when a speaker of interest has to be localized in a multiple speaker (“cocktail-party”) scenario. Forty-five healthy participants were tested, including younger (19–29 years; n = 21) and older (66–76 years; n = 24) age groups. Three conditions of short-term training (duration 15 min) were compared, requiring localization of non-speech targets under “cocktail-party” conditions with either (1) synchronous presentation of co-localized auditory-target and visual stimuli (audiovisual-congruency training) or (2) immediate visual feedback on correct or incorrect localization responses (visual-feedback training), or (3) presentation of spatially incongruent auditory-target and visual stimuli presented at random positions with synchronous onset (control condition). Prior to and after training, participants were tested in an auditory spatial attention task (15 min), requiring localization of a predefined spoken word out of three distractor words, which were presented with synchronous stimulus onset from different positions. Peaks of ERP components were analyzed with a specific focus on the N2, which is known to be a correlate of auditory selective spatial attention. N2 amplitudes were significantly larger after audiovisual-congruency training compared with the remaining training conditions for younger, but not older, participants. Also, at the time of the N2, distributed source analysis revealed an enhancement of neural activity induced by audiovisual-congruency training in dorsolateral prefrontal cortex (Brodmann area 9) for the younger group. These findings suggest that cross-modal processes induced by audiovisual-congruency training under “cocktail-party” conditions at a short time scale resulted in an enhancement of correlates of auditory selective spatial attention.

Younger and older adults show non-linear, stimulus-dependent performance during early stages of auditory training for non-native English.

Older adults often report difficulty understanding speech produced by non-native talkers. These listeners can achieve rapid adaptation to non-native speech, but few studies have assessed auditory training protocols to improve non-native speech recognition in older adults. In this study, a word-level training paradigm was employed, targeting improved recognition of Spanish-accented English. Younger and older adults were trained on Spanish-accented monosyllabic word pairs containing four phonemic contrasts (initial s/z, initial f/v, final b/p, final d/t) produced in English by multiple male native Spanish speakers. Listeners completed pre-testing, training, and post-testing over two sessions. Statistical methods, such as growth curve modeling and generalized additive mixed models, were employed to describe the patterns of rapid adaptation and how they varied between listener groups and phonemic contrasts. While the training protocol failed to elicit post-test improvements for recognition of Spanish-accented speech, examination of listeners' performance during the pre-testing period showed patterns of rapid adaptation that differed, depending on the nature of the phonemes to be learned and the listener group. Normal-hearing younger and older adults showed a faster rate of adaptation for non-native stimuli that were more nativelike in their productions, while older adults with hearing impairment did not realize this benefit.

Does industry funding differ between men and women in electrophysiology and devices research?

Abstract Funding Acknowledgements Type of funding sources: None. Background A significant gender bias has been identified in cardiology. Industry funding may be important developing electrophysiology (EP) and devices research. Such funding leads to potential conflicts of interest (COI) which must be disclosed on research presentations. There is limited evidence whether the gender bias in cardiology extends to industry funding of research. Purpose To assess whether COI disclosures in EP and devices presentations at the ESC Annual Congress differ between men and women. Methods Recorded presentations from the Arrhythmia & Devices section of ESC Annual Congresses 2016-2020 were assessed. Presentations were excluded if the original presentation contained no slides, it was a panel discussion, it was a non-scientific presentation, or part of the presentation was missing. Presentations with multiple speakers were also excluded. Chi-squared and Mann-Whitney U tests were used to assess differences between groups for dichotomous and continuous data respectively. Results Of 1,153 presentations assessed, 999 were suitable for inclusion. Women made up 22% (n = 221) of presenters. There was no difference in whether COI declaration slides were missing between women (5.9%, n = 13) and men (7.6%, n = 56), p = 0.38. In those with COI disclosure slides (n = 927), women declared significantly lower median number of COIs (0, IQR 0-3) compared with men (2, IQR 0-8), p < 0.0001. In contrast, women spent a greater time-per-word on their COI disclosure slides (250ms, IQR 125-375ms) compared with men (200ms, IQR 118-333ms), p < 0.0001. Conclusions Women made up a minority of presenters in EP/devices talks at the ESC annual congress. Women were less likely to have COIs which may suggest that they are less likely to receive industry funding. Despite this, women spent a greater amount of time-per-word on their COI slides. The lower number of declared COIs in women highlights another potential area of gender bias in cardiology that needs further investigation so that it can be addressed.

Is it all about the pitch? Acoustic determinants of dog-directed speech preference in domestic dogs, Canis familiaris

Dogs, similarly to infants, have been shown to be sensitive to human speech especially when it is directed to them. However, what essential acoustic, paralinguistic and lexical features of dog-directed speech are responsible for this preference in dogs is largely unknown. In the present study, generalized dog (DDS)-, infant (IDS)- and adult (ADS)-directed speech stimuli were created by using prerecorded sentences of multiple female speakers and these composite (averaged) stimuli were then manipulated to control for linguistic content as well as to equalize their mean fundamental frequency (F0) value. All three possible pairwise combinations of these acoustic stimuli were then presented to adult dogs in a two-way choice task where two identical target objects were used to indicate the sound sources. We found a significant preference towards the target object associated with DDS in the DDS versus ADS condition and suggest that, for dogs, mean F0 difference is not essential for DDS–ADS discrimination. However, we did not find evidence of selection bias when IDS was simultaneously presented either with DDS or ADS. Interestingly, our results also showed that dogs were more willing to approach the ‘more prosodic’ location (i.e. DDS or IDS versus ADS) when the prosodically more prominent sound stimulus was presented on their left side which suggests right-hemispheric specialization for neural processing of prosodic sounds in this domestic species. We also found that dogs made their choice faster when the ‘more prosodic’ stimulus was given first which suggests that they can perceive the difference not only between DDS and ADS, but also between IDS and ADS and between IDS and DDS. In conclusion, the composite DDS, IDS and ADS stimuli in the present study proved to be an effective technique in exploring the acoustic determinants of dog-directed speech preference in dogs.

Unsupervised Self-Adaptive Auditory Attention Decoding.

When multiple speakers talk simultaneously, a hearing device cannot identify which of these speakers the listener intends to attend to. Auditory attention decoding (AAD) algorithms can provide this information by, for example, reconstructing the attended speech envelope from electroencephalography (EEG) signals. However, these stimulus reconstruction decoders are traditionally trained in a supervised manner, requiring a dedicated training stage during which the attended speaker is known. Pre-trained subject-independent decoders alleviate the need of having such a per-user training stage but perform substantially worse than supervised subject-specific decoders that are tailored to the user. This motivates the development of a new unsupervised self-adapting training/updating procedure for a subject-specific decoder, which iteratively improves itself on unlabeled EEG data using its own predicted labels. This iterative updating procedure enables a self-leveraging effect, of which we provide a mathematical analysis that reveals the underlying mechanics. The proposed unsupervised algorithm, starting from a random decoder, results in a decoder that outperforms a supervised subject-independent decoder. Starting from a subject-independent decoder, the unsupervised algorithm even closely approximates the performance of a supervised subject-specific decoder. The developed unsupervised AAD algorithm thus combines the two advantages of a supervised subject-specific and subject-independent decoder: it approximates the performance of the former while retaining the 'plug-and-play' character of the latter. As the proposed algorithm can be used to automatically adapt to new users, as well as over time when new EEG data is being recorded, it contributes to more practical neuro-steered hearing devices.

Teleconferencing for Virtual Visiting Professors and Virtual Grand Rounds.

The 2020 global pandemic related to the coronavirus has led to unprecedented interruptions in typical patient care and resident education. Teleconferencing software was deployed by many institutions to comply with quarantine and social-distancing regulations. To supplement the loss of clinical experience for trainees, the authors implemented a novel virtual-educational programming using virtual visiting professors and virtual grand rounds. The authors describe the two different formats and advantages such as access to multiple speakers on diverse, innovative topics and decreased financial burdens to the host program. However, the authors do acknowledge some disadvantages from lack of face-to-face social interaction/networking and the need to consider time-zone differences. Both new programs were embraced by trainees at the authors' own institution and residents/medical students across the United States and Canada and around the world. The authors believe teleconferencing should be permanently incorporated into future educational opportunities for plastic surgeons, as it provides easy access to high-quality information.

Dynamically localizing multiple speakers based on the time-frequency domain

In this study, we present a deep neural network-based online multi-speaker localization algorithm based on a multi-microphone array. Following the W-disjoint orthogonality principle in the spectral domain, time-frequency (TF) bin is dominated by a single speaker and hence by a single direction of arrival (DOA). A fully convolutional network is trained with instantaneous spatial features to estimate the DOA for each TF bin. The high-resolution classification enables the network to accurately and simultaneously localize and track multiple speakers, both static and dynamic. Elaborated experimental study using simulated and real-life recordings in static and dynamic scenarios demonstrates that the proposed algorithm significantly outperforms both classic and recent deep-learning-based algorithms. Finally, as a byproduct, we further show that the proposed method is also capable of separating moving speakers by the application of the obtained TF masks.