Multiple Speakers Research Articles

Is 10 Better than 1? The Effect of Speaker Variability on Children’s Cross-Situational Word Learning

ABSTRACT The current study examined the effect of speaker variability on children’s cross-situational word learning (XSWL). The study also examined the role of bilingual experience and sustained attention. Forty English monolingual children and 40 Spanish-English bilingual children ages 4–7 completed a XSWL task in a Single Speaker Condition and a Multiple Speaker Condition. Results indicated that speaker variability neither facilitated nor hindered XSWL. While monolingual children outperformed bilingual children, speaker-variability effects did not fluctuate across the two language groups. Notably, exposure to multiple speakers facilitated XSWL in children with poorer sustained attention skills, suggesting that variability in the input may be especially useful to children with poorer cognitive processing abilities.

Lip Reading using CNN Lip Deflection Classifier and GAN Two-Stage Lip Corrector

Lip reading is a task to infer the content of speech through the movement of the lips, which is a technology with wide application prospects. Thanks to the development of deep learning technology and the proposal of large-scale datasets, lip reading has made great progress. However, these large-scale datasets contain only small angle lip deflection which is different with the wild environment. Most networks based on these datasets take some simple methods to correct the lip deflection that led to performance degradation when in the wild. In this work, we proposed a very challenging dataset with large angle lip deflection and multiple speakers to simulate the wild environment, and we designed a lip deflection classifier based on Convolutional Neural Network and a two-stage corrector based on Generative Adversarial Network. We use these two models to correct the lip deflection and improve the recognition accuracy. Our proposed method achieves an absolute improvement of 18.3% and 7.4%, respectively, compared with no preprocessing and just face alignment, which shows that our proposed network is more suitable to correct the large angle lip deflection in the wild.

Activities of the Right Temporo-Parieto-Occipital Junction Reflect Spatial Hearing Ability in Cochlear Implant Users

Spatial hearing is critical for us not only to orient ourselves in space, but also to follow a conversation with multiple speakers involved in a complex sound environment. The hearing ability of people who suffered from severe sensorineural hearing loss can be restored by cochlear implants (CIs), however, with a large outcome variability. Yet, the causes of the CI performance variability remain incompletely understood. Despite the CI-based restoration of the peripheral auditory input, central auditory processing might still not function fully. Here we developed a multi-modal repetition suppression (MMRS) paradigm that is capable of capturing stimulus property-specific processing, in order to identify the neural correlates of spatial hearing and potential central neural indexes useful for the rehabilitation of sound localization in CI users. To this end, 17 normal hearing and 13 CI participants underwent the MMRS task while their brain activity was recorded with a 256-channel electroencephalography (EEG). The participants were required to discriminate between the probe sound location coming from a horizontal array of loudspeakers. The EEG MMRS response following the probe sound was elicited at various brain regions and at different stages of processing. Interestingly, the more similar this differential MMRS response in the right temporo-parieto-occipital (TPO) junction in CI users was to the normal hearing group, the better was the spatial hearing performance in individual CI users. Based on this finding, we suggest that the differential MMRS response at the right TPO junction could serve as a central neural index for intact or impaired sound localization abilities.

NMF-weighted SRP for multi-speaker direction of arrival estimation: robustness to spatial aliasing while exploiting sparsity in the atom-time domain

Localization of multiple speakers using microphone arrays remains a challenging problem, especially in the presence of noise and reverberation. State-of-the-art localization algorithms generally exploit the sparsity of speech in some representation for this purpose. Whereas the broadband approaches exploit time-domain sparsity for multi-speaker localization, narrowband approaches can additionally exploit sparsity and disjointness in the time-frequency representation. Broadband approaches are robust to spatial aliasing but do not optimally exploit the frequency domain sparsity, leading to poor localization performance for arrays with short inter-microphone distances. Narrowband approaches, on the other hand, are vulnerable to spatial aliasing, making them unsuitable for arrays with large inter-microphone spacing. Proposed here is an approach that decomposes a signal spectrum into a weighted sum of broadband spectral components (atoms) and then exploits signal sparsity in the time-atom representation for simultaneous multiple source localization. The decomposition into atoms is performed in situ using non-negative matrix factorization (NMF) of the short-term amplitude spectra and the localization estimate is obtained via a broadband steered-response power (SRP) approach for each active atom of a time frame. This SRP-NMF approach thereby combines the advantages of the narrowband and broadband approaches and performs well on the multi-speaker localization task for a broad range of inter-microphone spacings. On tests conducted on real-world data from public challenges such as SiSEC and LOCATA, and on data generated from recorded room impulse responses, the SRP-NMF approach outperforms the commonly used variants of narrowband and broadband localization approaches in terms of source detection capability and localization accuracy.

Remote Intimacy

Remote Intimacy

Wave-domain active noise control over distributed networks of multi-channel nodes

Active noise control (ANC) in wave domain (spatial Fourier transform) can perform noise reduction within a large spatial region. Centralized algorithms collect signals at all error microphones for processing, which has a high computational complexity and is difficult to achieve large-scale deployment. In this work, we use multi-node networks to build the wave-domain ANC system, and propose a distributed algorithm to share computational burden between nodes. In particular, we consider a node as an unit consisting of multiple microphones, multiple speakers and a processor endowed with communication and computational capacity. Distributed algorithm is then specifically designed under this setting. Simulation results illustrate that the ANC system, equipped with the proposed distributed wave-domain ANC scheme, converges to the performance of the centralized solution.

Audio-Visual Tracking of Concurrent Speakers

Audio-visual tracking of an unknown number of concurrent speakers in 3D is a challenging task, especially when sound and video are collected with a compact sensing platform. In this paper, we propose a tracker that builds on generative and discriminative audio-visual likelihood models formulated in a particle filtering framework. We localize multiple concurrent speakers with a de-emphasized acoustic map assisted by the image detection-derived 3D video observations. The 3D multi-modal observations are either assigned to existing tracks for discriminative likelihood computation or used to initialize new tracks. The generative likelihoods rely on color distribution of the target and the de-emphasized acoustic map value. Experiments on AV16.3 and CAV3D datasets show that the proposed tracker outperforms the uni-modal trackers and the state-of-the-art approaches both in 3D and on the image plane.

WindTrack: Leveraging Sound and Wind to Track Angular Position of Users on Commercial Internet-Connected Fans

A recent trend in electric fans is the support of smarter features based on Internet connectivity, such as remote control. This work opens new opportunities to make these devices more intelligent, particularly by proposing WindTrack, a novel angular positioning technique for electric fans. This would encourage the emergence of a new class of smart fans that use spatial information for automatic wind direction control. Many works have extensively studied angular positioning, but had limitations realizing such smart fans in terms of deployability (due to hardware requirements), accuracy, and usability. WindTrack overcomes these limitations by leveraging the acoustic phenomenon of refraction caused by wind. More specifically, it transmits and receives sound signals by using a smartphone located close to a user, while operating a fan in an oscillating mode. Therefore, WindTrack only requires Internet-connected fans for collaboration with smartphones. It then identifies the user’s angular position based on the relationship between the propagation characteristics of the received signals and the oscillating fan. We further improve the robustness of WindTrack by using multiple speakers and microphones built-in smartphones and listening to wind sounds captured when wind passes across microphones. Extensive experiments on a WindTrack prototype demonstrate that it can achieve an angular positioning accuracy of a few degrees, especially in various environments and with no modification to commercial smartphones and Internet-connected fans.

A New Proposal for Phoneme Acquisition: Computing Speaker-Specific Distribution.

Speech is an acoustically variable signal, and one of the sources of this variation is the presence of multiple speakers. Empirical evidence has suggested that adult listeners possess remarkably sensitive (and systematic) abilities to process speech signals, despite speaker variability. It includes not only a sensitivity to speaker-specific variation, but also an ability to utilize speaker variation with other sources of information for further processing. Recently, many studies also showed that young children seem to possess a similar capacity. This suggests continuity in the processing of speaker-dependent speech variability, and suggests that this ability could also be important for infants learning their native language. In the present paper, we review evidence for speaker variability and speech processing in adults, and speaker variability and speech processing in young children, with an emphasis on how they make use of speaker-specific information in word learning situations. Finally, we will build on these findings to make a novel proposal for the use of speaker-specific information processing in phoneme learning in infancy.

Forward-backward recursive expectation-maximization for concurrent speaker tracking

In this paper, a study addressing the task of tracking multiple concurrent speakers in reverberant conditions is presented. Since both past and future observations can contribute to the current location estimate, we propose a forward-backward approach, which improves tracking accuracy by introducing near-future data to the estimator, in the cost of an additional short latency. Unlike classical target tracking, we apply a non-Bayesian approach, which does not make assumptions with respect to the target trajectories, except for assuming a realistic change in the parameters due to natural behaviour. The proposed method is based on the recursive expectation-maximization (REM) approach. The new method is dubbed forward-backward recursive expectation-maximization (FB-REM). The performance is demonstrated using an experimental study, where the tested scenarios involve both simulated and recorded signals, with typical reverberation levels and multiple moving sources. It is shown that the proposed algorithm outperforms the regular common causal (REM).

Many-to-Many Voice Transformer Network

This paper proposes a voice conversion (VC) method based on a sequence-to-sequence (S2S) learning framework, which enables simultaneous conversion of the voice characteristics, pitch contour, and duration of input speech. We previously proposed an S2S-based VC method using a transformer network architecture called the voice transformer network (VTN). The original VTN was designed to learn only a mapping of speech feature sequences from one speaker to another. Here, the main idea we propose is an extension of the original VTN that can simultaneously learn mappings among multiple speakers. This extension, called the many-to-many VTN, enables us to fully use available training data collected from multiple speakers by capturing common latent features that can be shared across different speakers. It also allows us to introduce a training loss called the identity mapping loss to ensure that the input feature sequence will remain unchanged when the source and target speaker indices are the same. Using this particular loss for model training has been found to be extremely effective in improving the performance of the model at test time. We conducted speaker identity conversion experiments and found that our model obtained higher sound quality and speaker similarity than baseline methods. We also found that our model, with a slight modification to its architecture, can handle any-to-many conversion tasks reasonably well.

Hearing Impaired Participants Improve More Under Envelope-Transcranial Alternating Current Stimulation When Signal to Noise Ratio Is High.

An issue commonly expressed by hearing aid users is a difficulty to understand speech in complex hearing scenarios, that is, when speech is presented together with background noise or in situations with multiple speakers. Conventional hearing aids are already designed with these issues in mind, using beamforming to only enhance sound from a specific direction, but these are limited in solving these issues as they can only modulate incoming sound at the cochlear level. However, evidence exists that age-related hearing loss might partially be caused later in the hearing processes due to brain processes slowing down and becoming less efficient. In this study, we tested whether it would be possible to improve the hearing process at the cortical level by improving neural tracking of speech. The speech envelopes of target sentences were transformed into an electrical signal and stimulated onto elderly participants’ cortices using transcranial alternating current stimulation (tACS). We compared 2 different signal to noise ratios (SNRs) with 5 different delays between sound presentation and stimulation ranging from 50 ms to 150 ms, and the differences in effects between elderly normal hearing and elderly hearing impaired participants. When the task was performed at a high SNR, hearing impaired participants appeared to gain more from envelope-tACS compared to when the task was performed at a lower SNR. This was not the case for normal hearing participants. Furthermore, a post-hoc analysis of the different time-lags suggest that elderly were significantly better at a stimulation time-lag of 150 ms when the task was presented at a high SNR. In this paper, we outline why these effects are worth exploring further, and what they tell us about the optimal tACS time-lag.

Learning Optimum Number of Bases for Indian Languages in Non-negative Matrix Factorization based Multilingual Speech Separation

Non-negative matrix factorization-based audio source separation separating a target source has shown significant performance improvement when the spectral bases attained after factorization exhibits latent structures in the mixed audio signal comprising multiple speaker sources. If all the sources are known, the spectral bases may be inferred on priority by using a training process on the database of isolated sources. The number of bases inferred for a source should not include bases matching spectral patterns of the interfering sources in the audio mixture; otherwise, the estimated target source after separation will be incorporated with undesirable spectral patterns. It is difficult to distinguish and separate similar audio sources in an overlapped speech, leading to a complex speech processing task. Therefore, this research attempts to learn an optimum number of bases for Indian languages leading to successful separation of target source in multi-lingual multiple speaker speech mixtures using non-negative matrix factorization. The languages used for utterances are Hindi, Marathi, Gujarati, and Bengali. The speaker combinations used are female-female, male-male, and female-male. The optimum number of bases which was determined by evaluating improvement in the separation performance was found to be 40 for all the languages considered.

A Study on the implementation of immersive sound using multiple speaker systems according to the location of sound sources in live performance

A Study on the implementation of immersive sound using multiple speaker systems according to the location of sound sources in live performance

Impact of ophthalmic webinars on the resident's learning experience during COVID-19 pandemic: An insight into its present and future prospects.

Purpose:To analyze the impact of ophthalmic webinars on the resident's learning experience during the COVID-19 pandemic (CP).Methods:This cross-sectional nationwide study was carried out for 1 month during CP and included a total of 382 ophthalmic residents. A questionnaire was sent through various social media platforms.Results:Residents expressed a decline in their clinical exposure (74%; 220), thesis work (58%; 218), and acquisition of the knowledge and skills (42.5%; 161) during CP. Benefits of webinars as perceived by the residents included gain in additional knowledge (77%; 286), feedback on queries (56%; 209), access to multiple speakers (50%; 191), and topics (30%; 110). Nearly 75% (291) of residents endorsed webinars as good to the very good academic tool, and 54% (202) preferred to continue attending webinars in the post-CP phase. However, connectivity/download/data issues (54%; 200) followed by loss of personal touch (53%; 188), lengthy or irrelevant topic (37%; 134), and poor transmission quality (33%; 121) were major deterrents against the webinar.Conclusion:The current study generated overall mixed responses from the ophthalmic postgraduate residents in favor of webinars. In the present format, webinars bear enormous potentials to supplement the traditional learning tools by providing uninterrupted learning experiences. However, they are still limited by their pedagogical and technical issues.

Distributed multiple speaker tracking based on time delay estimation in microphone array network

Multiple speaker tracking in distributed microphone array (DMA) network is a challenging task. A critical issue for multiple speaker scenarios is to distinguish the ambiguous observation and associate it to the corresponding speaker, especially under reverberant and noisy environments. To address the problem, a distributed multiple speaker tracking method based on time delay estimation in DMA is proposed in this study. Specifically, the time delay estimated by the generalised cross-correlation function is treated as an observation. In order to distinguish the observation for each speaker, the possible time delays, refer to as candidates, are extracted based on data association technique. Considering the ambient influence, a time delay estimation strategy is designed to calculate the time delay for each speaker from the candidates. Finally, only the reliable time delays in DMA are propagated throughout the whole network by diffusion fusion algorithm and used for updating the speakers' state within the distributed Kalman filter framework. The proposed approach can track multiple speakers successfully in a non-centralised manner under reverberant and noisy environments. Simulation results indicate that, compared with other methods, the proposed method can achieve a smaller root mean square error for multiple speaker tracking, especially in adverse conditions.

Insights into rapid adaptation to Spanish-accented English words in younger and older adults

Older adults often report difficulty understanding speech produced by non-native talkers. Although numerous studies have described auditory training protocols to improve speech recognition in noise, there are comparatively few studies evaluating auditory training for improving recognition of non-native speech. However, there is evidence that older adults can achieve short-term, rapid adaptation to non-native speech. In this study, a word-level training paradigm was employed, targeting improved recognition of Spanish-accented English. Younger and older adults with normal hearing or hearing impairment were trained on Spanish-accented monosyllabic word pairs containing four phonemic contrasts (initial s/z, initial v/f, final b/p, final d/t) produced in English by multiple male native Spanish speakers. Listeners completed pre-testing, training, and post-testing, over two sessions. While the training protocol failed to elicit any long-term learning for Spanish-accented speech, detailed examination of listeners' performance during the pre-testing revealed patterns of rapid adaptation that appear to vary by phonemic contrast and that may be dependent on the acoustic features of the stimulus. Statistical methods such as growth curve modeling and generalized additive mixed models are employed to describe the patterns of rapid adaptation, and how they vary between listener groups and different stimulus types.

Co-locating remote collaborators in immersive virtual environments using telematic systems

The rapid shift to remote work has forever changed the dynamic of teams, which require the best tools to work collaboratively across any distance. Rensselaer is home to two immersive virtual environments, intelligent rooms with panoramic, human-scale projection screens, spatial audio loudspeaker arrays, and networks of time-of-flight and acoustical tracking sensors. This project seeks to “colocate” teams across both sites such that the experience mimics collaborating within the same room. Simple video conferencing software are rarely well-configured for large groups and do not consider users’ spatial arrangements. This approach captures ultra-low-latency video and audio feeds of each space for presentation at either end, enabling a group at each location to communicate at 1-to-1 scale with the other. A spherical microphone array tracks multiple simultaneous speakers and adjusts their spatial positions across a Wave Field Synthesis loudspeaker array, maintaining audio-visual congruency. Users at each site can use the panoramic displays to present panoramic imagery or immersive data to be explored simultaneously by each group, facilitating a collaborative immersive experience, and interactions with the screen at one site may be echoed at the other to create the appearance of a single space. [Work supported by CISL, NSF No. 1229391, Army DURIP No. 68604-CS-RIP.]

Pragmatic Markers in the Aspect of Communicative Alignmen

The article presents a model of communicative alignment in pragmatic markers (PM) use in Russian everyday dialogical communication. The main objectives are to check whether speakers coordinate their linguistic behavior not just with the use of lexemes or grammar forms or constructions, but also with PMs and how this actually works. We suppose that the use of PM by one of the speakers in the dialogue may increase the chances that the same PM will be used by the other speaker. In a Russian speech corpus "One Day of Speech" there were found 57 macro-episodes of communication where the PM koroche / koroche govorya was used by multiple speakers (46 episodes with 2 speakers using the PM, 11 – with 3 speakers). The analysis of PM use applying the system of quantitative parameters, worked out by the authors, has enabled them to note, that medium frequency of PM use rises when the number of speakers increases. PM used by speaker 1 is repeated by speaker 2, thus inducing speaker 3 to use the same PM, which influences the speech of the first two speakers respectively. The data analysis allows us to conclude that the original hypothesis of alignment in PMs has been proved for the marker we studied.

Paying attention to speech: The role of working memory capacity and professional experience.

Managing attention in multispeaker environments is a challenging feat that is critical for human performance. However, why some people are better than others in allocating attention appropriately remains highly unknown. Here, we investigated the contribution of two factors-working memory capacity (WMC) and professional experience-to performance on two different types of attention task: selective attention to one speaker and distributed attention among multiple concurrent speakers. We compared performance across three groups: individuals with low (n = 20) and high (n = 25) WMC, and aircraft pilots (n = 24), whose profession poses extremely high demands for both selective and distributed attention to speech. Results suggests that selective attention is highly effective, with good performance maintained under increasingly adverse conditions, whereas performance decreases substantially with the requirement to distribute attention among a larger number of speakers. Importantly, both types of attention benefit from higher WMC, suggesting reliance on some common capacity-limited resources. However, only selective attention was further improved in the pilots, pointing to its flexible and trainable nature, whereas distributed attention seems to suffer from more fixed and severe processing bottlenecks.