Pairs Of Speakers Research Articles

Zygosity Effects on Human Voice: Fundamental Frequency Analysis of Brazilian Twins' Speech.

Voice production can be influenced by interindividual variations related to genetic, physiological, behavioral, and several environmental factors. Here we examined the effect of zygosity on speaking fundamental frequency (F0) statistical descriptors. Our aims were: (1) to determine whether the genetic similarity between monozygotic (MZ) and dizygotic (DZ) twins affects F0 characteristics, and (2) to quantify the contribution of genetic factors to these characteristics. The study involved 79 same-sex twin pairs of Brazilian Portuguese speakers, comprising 65 MZ and 14 DZ twins, aged 18 to 66 years (31.7 ± 11.6 years), with 21 male and 58 female pairs. Participants were recorded while uttering a greeting phrase and the Brazilian Portuguese version of the 'Happy Birthday to You' song. Speech segments were analyzed using Praat free software, and F0 measures were automatically extracted in both Hertz and semitone scales. Statistical descriptors, including centrality, dispersion, and extreme values of F0 were examined, and the ACE model (i.e., total genetic effects, A; shared environmental influences, C; and nonshared environmental influences, E) was employed to estimate the additive effect;ts of monozygosity. As anticipated, we observed a zygosity effect on several F0 parameters, with more similarity between MZ twins compared to DZ twins. We discuss the genetic influences on F0 parameters and the absence of a monozygosity effect in two of them. Additionally, we briefly address potential biases associated with the selected measurement scale for statistical modeling. Finally, we explore the influence of genetic factors on F0 patterns, as well as environmental, life history and linguistic factors, particularly concerning F0 variation in speech.

EyeGesener: Eye Gesture Listener for Smart Glasses Interaction Using Acoustic Sensing

The smart glasses market has witnessed significant growth in recent years. The interaction of commercial smart glasses mostly relies on the hand, which is unsuitable for scenarios where both hands are occupied. In this paper, we propose EyeGesener, an eye gesture listener for smart glasses interaction using acoustic sensing. To mitigate the Midas touch problem, we meticulously design eye gestures for interaction as two intentional consecutive saccades in a specific direction without visual dwell. The proposed system is a glass-mounted acoustic sensing system with two pairs of commercial speakers and microphones to sense eye gestures. To capture the subtle movements of the eyelid and surrounding skin induced by eye gestures, we design an Orthogonal Frequency Division Multiplexing (OFDM)-based channel impulse response (CIR) estimation schema that allows two speakers to transmit at the same time and in the same frequency band without collision. We implement eye gesture filtering and adversarial-based eye gesture recognition to identify eye gestures for interaction, filtering out daily eye movements. To address the differences in eye size and facial structure among different users, we employ adversarial training to achieve user-independent eye gesture recognition. We evaluate the performance of our system through experiments on data collected from 16 subjects. The experimental result shows that our system can recognize eight eye gestures with an average F1-score of 0.93, and the false alarm rate of our system is 0.03. We develop an interactive real-time audio-video player based on EyeGesener and then conduct a user study. The result demonstrates the high usability of the proposed system.

OmniResMonitor: Omnimonitoring of Human Respiration using Acoustic Multipath Reflection

Contactless respiration monitoring using wireless signals has drawn much attention in recent years. Many approaches have been proposed, however, they may not work when there is a lack of signals directly reflected from target's chest, e.g., a target faces away from the transceiver or a target is blocked by furniture. In this paper, we design and implement a novel omnimonitoring system for human respiration, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">OmniRespMonitor</i> , using a pair of speaker and microphone. Different from Radio Frequency (RF) signal, acoustic signals cannot penetrate through walls and furniture. The multipath reflection in an indoor environment will result in highly abundant acoustic signals. In this case, even though there are lack of acoustic signals directly reflected by a target's chest, indirectly-reflected acoustic signals can still be received by the microphone. We can therefore monitor the target's respiration by extracting this subtle variation of indirectly reflected signals. To achieve this, we model chest movement using truncated System Frequency Response (SFR). We then develop a global search method based on the autocorrelation function to extract minute chest movement from SFR sequences. Finally, we dynamically synthesize the chest movement information to recover the breathing wave in real time. We conduct extensive experiments with both humans and animals (goat), the results show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">OmniResMonitor</i> is able to monitor single target's respiration within 5 meters in indoor environments in various challenging scenarios there are lack of directly-reflected acoustic signals.

AudioGuard: Omnidirectional Indoor Intrusion Detection Using Audio Device

Indoor intrusion detection is a critical task for home security. Previous works in intrusion detection suffer from the problems such as blind spots in non-line-of-sight (NLOS) areas, restricted device locations, massive offline training required, and privacy concern. In this article, we design and implement an omnidirectional indoor intrusion detection system, named AudioGuard , using only a pair of speaker and microphone. AudioGuard is able to detect both line-of-sight (LOS) and NLOS intrusions. Our observation of acoustic signal propagation in an indoor environment shows that there exist abundant multipath reflections and human movement introduces Doppler shift in echo signals. We hence capture periodical Doppler shift caused by intruder's walking motion to detect intrusion. Specifically, we first extract the Doppler shift embedded in echo signals, and we then propose a periodicity polarization method to cancel out the impact of the change of radial angle and the distance on periodicity of Doppler shift. Finally, we detect intrusion by measuring periodicity of Doppler shift over time. Extensive experiments show that AudioGuard achieves a miss report rate of 0% and 1.75% for LOS and NLOS intrusion, respectively, and a false alarm rate of 4.17%.

Design and characterisation of a 3-D microphone array for windtunnel measurements

A three-dimensional (3-D) array of GRAS 40PH-10 CCP Free-field Array Microphones has been designed for use in the UNSW anechoic wind tunnel (UAT) for aeroacoustic measurements. The array consists of 192 microphones split into three identical planar arrays of 64 microphones which are placed on both sides and above the potential core of the open-jet that issues into the wind tunnel test section. The planar arrays feature an Underbrink spiral comprised of eight arms of eight microphones each. The spiral configuration was selected to maximize signal-to-noise ratio (SNR) and minimize the beamwidth given the geometrical restrictions of the UAT. A white noise speaker was placed at a range of locations within the testing domain to characterize the source localization abilities of the array as a function of position and frequency. Furthermore, a pair of independent white noise speakers were placed at various separation distances to determine the array’s capability to identify multiple sources within the domain. A 3-D implementation of conventional cross-spectral beamforming and Clean-SC deconvolution algorithms was used, and the performance of each was compared. It was determined that Clean-SC was the preferred method for 3-D beamforming in terms of source localization accuracy, SNR, and beamwidth, despite the higher computational cost.

PoseSonic

In this paper, we introduce PoseSonic, an intelligent acoustic sensing solution for smartglasses that estimates upper body poses. Our system only requires two pairs of microphones and speakers on the hinges of the eyeglasses to emit FMCW-encoded inaudible acoustic signals and receive reflected signals for body pose estimation. Using a customized deep learning model, PoseSonic estimates the 3D positions of 9 body joints including the shoulders, elbows, wrists, hips, and nose. We adopt a cross-modal supervision strategy to train our model using synchronized RGB video frames as ground truth. We conducted in-lab and semi-in-the-wild user studies with 22 participants to evaluate PoseSonic, and our user-independent model achieved a mean per joint position error of 6.17 cm in the lab setting and 14.12 cm in semi-in-the-wild setting when predicting the 9 body joint positions in 3D. Our further studies show that the performance was not significantly impacted by different surroundings or when the devices were remounted or by real-world environmental noise. Finally, we discuss the opportunities, challenges, and limitations of deploying PoseSonic in real-world applications.

The Curious Case of Impersonators and Singers: Telling Voices Apart and Telling Voices Together under Naturally Challenging Listening Conditions.

Vocal identity processing depends on the ability to tell apart two instances of different speakers whilst also being able to tell together two instances of the same speaker. Whilst previous research has examined these voice processing capabilities under relatively common listening conditions, it has not yet tested the limits of these capabilities. Here, two studies are presented that employ challenging listening tasks to determine just how good we are at these voice processing tasks. In Experiment 1, 54 university students were asked to distinguish between very similar sounding, yet different speakers (celebrity targets and their impersonators). Participants completed a 'Same/Different' task and a 'Which is the Celebrity?' task to pairs of speakers, and a 'Real or Not?' task to individual speakers. In Experiment 2, a separate group of 40 university students was asked to pair very different sounding instances of the same speakers (speaking and singing). Participants were presented with an array of voice clips and completed a 'Pairs Task' as a variant of the more traditional voice sorting task. The results of Experiment 1 suggested that significantly more mistakes were made when distinguishing celebrity targets from their impersonators than when distinguishing the same targets from control voices. Nevertheless, listeners were significantly better than chance in all three tasks despite the challenge. Similarly, the results of Experiment 2 suggested that it was significantly more difficult to pair singing and speaking clips than to pair two speaking clips, particularly when the speakers were unfamiliar. Again, however, the performance was significantly above zero, and was again better than chance in a cautious comparison. Taken together, the results suggest that vocal identity processing is a highly adaptable task, assisted by familiarity with the speaker. However, the fact that performance remained above chance in all tasks suggests that we had not reached the limit of our listeners' capability, despite the considerable listening challenges introduced. We conclude that voice processing is far better than previous research might have presumed.

Study on an evaluation method of virus exposure risk by droplets focusing on a sound environment

The final goal of our study is to propose a new method to evaluate the risk of virus infection from the aspect of the sound environment. In this study, we attempted to quantitatively grasp the relationship between speech level and number of droplets when talking while eating at a restaurant. A psychoacoustic experiment was conducted in which the subject was made to speak under the condition that the background noise level was varied every 5 dB from 55 dB to 70 dB using a pair of speaker and listener as subject. From the results, the effect of background noise level on speech level was examined, and the relationship between speech level and number of droplets was clarified. Although the speech level tended to increase with increase in the background noise level, in the background noise range of 55 dB to 70 dB, there was no significant difference in the number of droplets with particle sizes larger than 100 μm. Furthermore, we evaluated the risk of virus exposure in droplet infection focusing on the indoor background noise level.

Vowel convergence does not affect auditory speaker discriminability in humans and machine in a case study on Swiss German dialects

In this study, we examined whether the convergence in interlocutors’ vowel acoustics leads to decreasing discriminability between interlocutors’ voices. Ten pairs of Grison and Zürich German speakers produced lexical items before and after dialogue interactions with evidence of vowel convergence in post-dialogue productions. In Experiment 1, native and non-native Swiss German listeners discriminated pairs of speakers whose speech was obtained pre- and post-dialogue. Results showed that listeners’ sensitivity (A’) was higher for native than non-native listeners, but comparable for pre- and post-dialogue recordings. The observed negative correlation between voice discrimination and the acoustic distance in formant space was mainly driven by a single speaker pair. In Experiment 2, the speaker recognition performance of an i-vector-based software was compared in pre- and post-dialogue speech. Results revealed no difference in the system performance between the two conditions. The findings suggest that vowel convergence does not compromise voice discriminability under the given experimental conditions.

A Comparative Study of Topic Selection Schema in The First-time Conversation Between Japanese and Sundanese Native Speakers

First-time conversations play an important role as a starting point in building relationships. However, the lack of information about interlocutors makes it difficult to decide what topic to take up. In this study, we targeted 10 pairs of native Japanese speakers and Sundanese native speakers, clarified what topics would be selected in the first-time conversation by university students, and examined similarities and differences between both native speakers. As a result, in the conversation data of the Japanese pairs, 20 topics out of 83 topics were recognized as “topic items”, and depending on the relationship between the topic items, they were then can be classified into eight categories, namely “affiliation”, “origin”, “university life”, “hobbies/enjoyment”, “living”, “commonalities”, “specialty”, and “society”. On the other hand, in the conversation data of the Sundanese pairs, 16 topics out of 95 topics were recognized as “topic items”, and they can be classified into seven categories, namely “university life”, “affiliation”, “residential”, “origin”, “commonalities”, “specialty”, and “society”. The overall picture of the classified categories and the topic items corresponding to the subclasses is the “topic selection list”, and the set of culturally shared knowledge about this “list” is the “first-time conversation topic selection schema”. The results of this study can be applied as a reference for topic selection in the first conversations with Japanese native speakers or Sundanese native speakers, especially between university students. The results also can be used as a repository of scientific knowledge in related fields such as sociolinguistic studies on conversational analysis, and as a reference on Japanese language education studies in general.

Children Treat Grammatical Errors Differently for Native and Non-Native Speakers.

Both children and adults demonstrate biases against non-native speakers. However, in some situations, adults act more generously towards non-native speakers than towards native speakers. In particular, adults judge errors from non-native speakers less harshly, presumably because they expect such errors. In the present study, we asked whether 5-6-year-old children place less weight on errors from speakers with a foreign accent. In Experiment 1, 5- and 6-year-old children (N = 80) listened to pairs of either native or foreign-accented speakers (between-subjects) label objects. For native speaker pairings, children preferred information provided by grammatical speakers over information from speakers who made subject-verb agreement errors. In contrast, children chose between foreign-accented speakers at chance. In Experiment 2 (N = 40), children preferred information from grammatical foreign-accented speakers over information from foreign-accented speakers who produced word-order violations. These findings constitute the first demonstration that children treat speech errors differently based on a speaker’s language background.

LASense: Pushing the Limits of Fine-grained Activity Sensing Using Acoustic Signals.

Acoustic signals have been widely adopted in sensing fine-grained human activities, including respiration monitoring, finger tracking, eye blink detection, etc. One major challenge for acoustic sensing is the extremely limited sensing range, which becomes even more severe when sensing fine-grained activities. Different from the prior efforts that adopt multiple microphones and/or advanced deep learning techniques for long sensing range, we propose a system called LASense, which can significantly increase the sensing range for fine-grained human activities using a single pair of speaker and microphone. To achieve this, LASense introduces a virtual transceiver idea that purely leverages delicate signal processing techniques in software. To demonstrate the effectiveness of LASense, we apply the proposed approach to three fine-grained human activities, i.e., respiration, finger tapping and eye blink. For respiration monitoring, we significantly increase the sensing range from the state-of-the-art 2 m to 6 m. For finer-grained finger tapping and eye blink detection, we increase the state-of-the-art sensing range by 150% and 80%, respectively.

Representation of others' synchronous and asynchronous sentences interferes with sentence production.

In dialogue, people represent each other's utterances to take turns and communicate successfully. In previous work, speakers who were naming single pictures or picture pairs represented whether another speaker was engaged in the same task (vs a different or no task) concurrently but did not represent in detail the content of the other speaker's utterance. Here, we investigate the co-representation of whole sentences. In three experiments, pairs of speakers imagined each other producing active or passive descriptions of transitive events. Speakers took longer to begin speaking when they believed their partner was also preparing to speak, compared to when they did not. Interference occurred when speakers believed their partners were preparing to speak at the same time as them (synchronous production and co-representation; Experiment 1), and also when speakers believed that their partner would speak only after them (asynchronous production and co-representation; Experiments 2a and 2b). However, interference was generally no greater when speakers believed their partner was preparing a different compared to a similar utterance, providing no consistent evidence that speakers represented what their partners were preparing to say. Taken together, these findings indicate that speakers can represent another's intention to speak even as they are themselves preparing to speak, but that such representation tends to lack detail.

Determining the Likelihood Ratio From Perceptual Attributes of Voice

To provide voice experts with a method for determining the likelihood ratio (LR) from the perceptual evaluation of distinctive voice attribute scores. The proposed method aims to obtain the similarity and typicality judgments made by forensic voice experts (FVEs) during the comparison of attributes in voice pairs. It is based on the scoring method for LR calculation. In the first stage, 17 perceptual attributes grouped into six vocalic categories are specified. A novel graphical interface is used to obtain discriminative responses both globally and for each attribute from ten pairs of test sentences produced by the same and different speakers. The FVEs should discriminate whether the attributes are similar or different in each pair and should indicate the degree to which the attributes are present. In addition, for six specific attributes, the FVEs must decide whether the attribute is typical or atypical in the reference population. In the second stage, the mean score obtained in the first stage is converted to LR using probability density functions of listeners' responses to 1680 same/different speaker pairs discriminated for female and male speakers. The responses of the FVEs to the test pairs show the discriminatory power of the attributes, the incidence of the typicality factor on the final score and the performance of each FVE. With the application of the probability density functions obtained for the responses to pairs of the same or different origin taken from the reference population, the final scores are converted into LRs that are compared with the true conditions of each pair. The application of the developed system allows the global and discriminated evaluation of the perceptual attributes with high agreement in the comparison of pairs of voices. Obtaining the LRs allows associating the perceptual evaluation method with the automatic methods that are used nowadays. The responses of the FVEs taken as a reference, will allow training and evaluating the performance of young FVEs.

SonicFace

Accurate recognition of facial expressions and emotional gestures is promising to understand the audience's feedback and engagement on the entertainment content. Existing methods are primarily based on various cameras or wearable sensors, which either raise privacy concerns or demand extra devices. To this aim, we propose a novel ubiquitous sensing system based on the commodity microphone array --- SonicFace, which provides an accessible, unobtrusive, contact-free, and privacy-preserving solution to monitor the user's emotional expressions continuously without playing hearable sound. SonicFace utilizes a pair of speaker and microphone array to recognize various fine-grained facial expressions and emotional hand gestures by emitted ultrasound and received echoes. Based on a set of experimental evaluations, the accuracy of recognizing 6 common facial expressions and 4 emotional gestures can reach around 80%. Besides, the extensive system evaluations with distinct configurations and an extended real-life case study have demonstrated the robustness and generalizability of the proposed SonicFace system.

Interference of Two Point Sources Using Smartphones

A simple experimental setup using a smartphone and a pair of speakers is presented to perform an accurate experiment on interference of two point sources. The proposed experiment allows simple but interesting measurements to be done to introduce students to interference and diffraction phenomena. As such, the experiment effectively introduces the classical Young’s double-slit interference experiment, and can be used as a replacement for the classical ripple tank.

Dialect contact in real interactions and in an agent-based model

This study investigates if and to what extent speakers change their vowel production after interacting with someone from a different dialect region. Phonetic accommodation was investigated for 14 mixed-dialect pairs in Swiss German. Vowels for each speaker pair were analysed acoustically before and after they interacted for about 20 minutes. Agent-based modelling was used to simulate the change in absence of social and interactional factors. The simulation predicted that convergence was not symmetric but depended on a vowel's variability and its orientation in the acoustic space prior to the interaction. No evidence of convergence was found for the accommodation study involving 14 real interactions. The lack of convergence is explained with reference to the specific sociolinguistic situation of German-speaking Switzerland, where dialects carry a high prestige and are mutually intelligible, and where mixed-dialect interactions are the norm.

Cover Art

Cover Art

Acoustic-based Upper Facial Action Recognition for Smart Eyewear

Smart eyewear (e.g., AR glasses) is considered to be the next big breakthrough for wearable devices. The interaction of state-of-the-art smart eyewear mostly relies on the touchpad which is obtrusive and not user-friendly. In this work, we propose a novel acoustic-based upper facial action (UFA) recognition system that serves as a hands-free interaction mechanism for smart eyewear. The proposed system is a glass-mounted acoustic sensing system with several pairs of commercial speakers and microphones to sense UFAs. There are two main challenges in designing the system. The first challenge is that the system is in a severe multipath environment and the received signal could have large attenuation due to the frequency-selective fading which will degrade the system's performance. To overcome this challenge, we design an Orthogonal Frequency Division Multiplexing (OFDM)-based channel state information (CSI) estimation scheme that is able to measure the phase changes caused by a facial action while mitigating the frequency-selective fading. The second challenge is that because the skin deformation caused by a facial action is tiny, the received signal has very small variations. Thus, it is hard to derive useful information directly from the received signal. To resolve this challenge, we apply a time-frequency analysis to derive the time-frequency domain signal from the CSI. We show that the derived time-frequency domain signal contains distinct patterns for different UFAs. Furthermore, we design a Convolutional Neural Network (CNN) to extract high-level features from the time-frequency patterns and classify the features into six UFAs, namely, cheek-raiser, brow-raiser, brow-lower, wink, blink and neutral. We evaluate the performance of our system through experiments on data collected from 26 subjects. The experimental result shows that our system can recognize the six UFAs with an average F1-score of 0.92.

DDRel: A New Dataset for Interpersonal Relation Classification in Dyadic Dialogues

Interpersonal language style shifting in dialogues is an interesting and almost instinctive ability of human. Understanding interpersonal relationship from language content is also a crucial step toward further understanding dialogues. Previous work mainly focuses on relation extraction between named entities in texts or within a single dialogue session. In this paper, we propose the task of relation classification of interlocutors based on their dialogues. We crawled movie scripts from IMSDb, and annotated the relation label for each session according to 13 pre-defined relationships. The annotated dataset DDRel consists of 6,300 dyadic dialogue sessions between 694 pairs of speakers with 53,126 utterances in total. We also construct session-level and pair-level relation classification tasks with widely-accepted baselines. The experimental results show that both tasks are challenging for existing models and the dataset will be useful for future research.