Individual Speaker Characteristics Research Articles

Speaker diarization with variants of self-attention and joint speaker embedding extractor

The speaker diarization task pertains to the automated differentiation of speakers within an audio recording, while lacking any prior information regarding the speakers. The introduction of the self-attention mechanism in End-to-End Neural Speaker Diarization (EEND) has elegantly resolved the issue of overlapping speakers. The Transformer model equipped with self-attention mechanism has shown great potential in collecting global information, yielding remarkable outcomes in various tasks. However, the individual speaker characteristics are predominantly reflected in the contextual information, which conventional self-attention would not adequately address. In this study, we propose a hierarchical encoders model to augment the encoders’ acquisition of speaker information in two distinct ways: (1) Constraining the perceptual field of the self-attentive mechanism with left-right windows or Gaussian weights to highlight contextual information; (2) Utilizing a pre-trained time-delay neural network based speaker embedding extractor to alleviate the shortcomings of speaker feature extraction ability. We evaluate the proposed methods on a simulated dataset of two speakers and a real conversation dataset. The model with the most favorable outcomes among the proposed enhancements achieves a diarization error rate of 7.74% on the simulated dataset and 21.92% on MagicData-RAMC after adaptation. These results compellingly demonstrate the efficacy of the proposed methods.

Speaker anonymization by modifying fundamental frequency and x-vector singular value

Speaker anonymization is a method of protecting voice privacy by concealing individual speaker characteristics while preserving linguistic information. The VoicePrivacy Challenge 2020 was initiated to generalize the task of speaker anonymization. In the challenge, two frameworks for speaker anonymization were introduced; in this study, we propose a method of improving the primary framework by modifying the state-of-the-art speaker individuality feature (namely, x-vector) in a neural waveform speech synthesis model. Our proposed method is constructed based on x-vector singular value modification with a clustering model. We also propose a technique of modifying the fundamental frequency and speech duration to enhance the anonymization performance. To evaluate our method, we carried out objective and subjective tests. The overall objective test results show that our proposed method improves the anonymization performance in terms of the speaker verifiability, whereas the subjective evaluation results show improvement in terms of the speaker dissimilarity. The intelligibility and naturalness of the anonymized speech with speech prosody modification were slightly reduced (less than 5% of word error rate) compared to the results obtained by the baseline system.

Source versus spectral cues in the perception of indexical features in speech

Spoken language includes two different forms of information: linguistic (message related) and indexical (related to individual speaker characteristics). This study explores the nature of the acoustic cues that listeners may use to identify gender and dialect. Spontaneous utterances were produced by 40 speakers (20 male, 20 female) from two different regional dialects spoken in central Ohio (OH) and western North Carolina (NC). These utterances were equally divided into three sets of tokens. One set was unprocessed (except for amplitude equalization). A second set was low-pass filtered at 400 Hz, retaining prosodic information, but little content. A third set was processed through an 8-channel noise vocoder eliminating all harmonic information (simulating cochlear implant processing). These three stimulus sets (blocked and randomized) were played to 20 OH listeners who identified whether the token was produced by a man or a woman, from OH or NC. Gender identification rates were high (means > 89%) across all three conditions with clear > LP filtered > vocoded. The rates for dialect identification were significantly lower overall with the LP-filtered condition close to chance (58%). Discussion will center on listener use of acoustic features and perceptual sensitivity (d’) to gender and dialect.

Frequency Warping for Speaker Adaptation in HMM-based Speech Synthesis

Speaker adaptation in speech synthesis transforms a source utterance to a target utterance that differs from the source in terms of voice characteristics. In this paper, we employ vocal tract length normalization, which is generally used in speech recognition to remove individual speaker characteristics, to speaker adaptation in speech synthesis. We propose a frequency warping approach based on a time-varying bilinear function to reduce the weighted spectral distance between the source speaker and the target speaker. The warped spectra of the source speaker are then converted to line spectrum pairs to train hidden Markov models (HMM). HMMs are further adapted by algorithms based on maximum likelihood linear regression with the target speaker's data. The experimental results show that our frequency warping approach can make the warped spectra of the source speaker closer to the target speaker, and the resultant adapted HMMs perform better than the HMMs trained by unwrapped spectra in terms of synthesized speech naturalness and speaker similarity.

Detection of speaker individual information using a phoneme effect suppression method

Feature extraction of speaker information from speech signals is a key procedure for exploring individual speaker characteristics and also the most critical part in a speaker recognition system, which needs to preserve individual information while attenuating linguistic information. However, it is difficult to separate individual from linguistic information in a given utterance. For this reason, we investigated a number of potential effects on speaker individual information that arise from differences in articulation due to speaker-specific morphology of the speech organs, comparing English, Chinese and Korean. We found that voiced and unvoiced phonemes have different frequency distributions in speaker information and these effects are consistent across the three languages, while the effect of nasal sounds on speaker individuality is language dependent. Because these differences are confounded with speaker individual information, feature extraction is negatively affected. Accordingly, a new feature extraction method is proposed to more accurately detect speaker individual information by suppressing phoneme-related effects, where the phoneme alignment is required once in constructing a filter bank for phoneme effect suppression, but is not necessary in processing feature extraction. The proposed method was evaluated by implementing it in GMM speaker models for speaker identification experiments. It is shown that the proposed approach outperformed both Mel Frequency Cepstrum Coefficient (MFCC) and the traditional F-ratio (FFCC). The use of the proposed feature has reduced recognition errors by 32.1–67.3% for the three languages compared with MFCC, and by 6.6–31% compared with FFCC. When combining an automatic phoneme aligner with the proposed method, the result demonstrated that the proposed method can detect speaker individuality with about the same accuracy as that based on manual phoneme alignment.

Audiovisual perception in adverse conditions: Language, speaker and listener effects

This study investigated the relative contribution of auditory and visual information to speech perception by looking at the effect of visual and auditory degradation on the weighting given to visual cues for native and non-native speakers. Multiple iterations of /ba/, /da/ and /ga/ by five Australian English and five Mandarin Chinese speakers were presented to Australian English, British English and Mandarin Chinese participants. Tokens were presented in auditory, visual and congruent/incongruent audiovisual (AV) modes, either in clear or with visual degradation (blurring), auditory degradation (noise) or combined degradations. In the AV clear condition, English-speaking participants showed greater visual weighting for non-native speakers, but this was not found for Chinese participants. In ‘single-channel degradation’ conditions, the weighting of the intact channel increased significantly, with little influence of speaker language. There was no strong evidence of native-language effects on the weighting of visual cues. The degree of visual weighting varied widely across individual participants, and was also affected by individual speaker characteristics. The weighting of auditory and visual cues is therefore highly flexible and dependent on the information load of each channel; non-native speaker and language-background effects may influence visual weighting but individual perceiver and speaker strategies also have a strong impact.

Language and dialect identification by syllabic spectral features

A speaker-based automatic language identification system has been studied to discriminate both languages and dialects. Syllabic spectral features are extracted automatically by an artificial neural network. The system makes no use of phonetic segmentation for recognition or training. High performance has been obtained for monologue telephone recordings. The system has been applied to several different databases to study the discrimination possible among different languages, dialects, and accents with such limited phonetic–acoustic information. The system performance is also related to individual speaker characteristics and the size of reference population for each language, and the length of testing samples. This study will report experimental results related to performance and data environments. It will also report an attempt by others to compare recognition performance of trained or expert human listeners with automatic recognition results.

Differences in speaker’s articulatory space: Their contribution to vowel gesture and acoustic pattern

The view that individual speaker characteristics reflect morphological and functional differences in the speech apparatus has been accepted as self-evident despite the lack of convincing empirical data. A recent study on the issue of individual differences based on x-ray microbeam data from ten American English and ten Japanese speakers (five males and five females in each group) has pointed to a significance of morphology in accounting for speaker characteristics of vowel gesture. The midsagittal form of a speaker’s orofacial structure was parametrized by several landmarks on x-ray scan images. A few points and lines were chosen to form a quadrilateral framework so that they define a geometrical ‘‘articulatory space’’ where articulation of the tongue–jaw complex takes place. Pellet data for vowels were mapped onto speaker’s articulatory space to observe the effect of morphological difference on vowel articulation. Preliminary results have shown the extent of morphological variation of the speech apparatus and adaptability of vowel gesture to different forms of the space. The aspect ratio of the articulatory space was judged as the primary causal factor of articulatory variation. Acoustic manifestation and perceptual contribution of the variation will be discussed from a viewpoint of auditory-articulatory linkage.

An approach to the problem of regional accent in automatic speech recognition

An approach to the problem of inter-speaker variability in automatic speech recognition is described which exploits systematic vowel differences in a two-stage process of adaptation to individual speaker characteristics. In stage one, an accent identification procedure selects one of four gross regional English accents on the basis of vowel quality differences within four calibration sentences. In stage two, an adjustment procedure shifts the regional reference vowel space onto the speaker's vowel space as calculated from the accent identification data. Results for 58 speakers from the four regional accent areas are presented.

Computer simulation of the fitting of hearing aids based on prediction of the discrimination ability of speech sounds

The earlier version of the computer program for simulating the process of fitting hearing aids [S. Hiki and Y. Fukuda, “Prediction of the ability of discrimination of speech sounds by hearing impaired children,” J. Acoust. Soc. Am. Suppl. 1 68, S5 (1980)] has been modified so that correction for individual speaker differences can be made. The latest version of the program consists of two parts. In the first part, by inputting various characteristics of the hearing loss of an impaired child and the candidate hearing aid, parameters of the hearing aid are controlled. The goal of the adjustment is to match the sound pressure versus frequency range, audible for each impaired child with the aided characteristics, to that of typical speech sounds. In the second part, the type of identifiable phones and probable phonetic confusions among the phones are estimated by analyzing the relationship between the aided characteristics and the acoustical properties of the speech sounds. The corrections for individual speaker characteristics involve the average amount of shift of formant frequencies for all vowels due to the differences of shape of the vocal tract, the deviation of average value of the fundamental frequency, and the deviation of intensity of the unvoiced sounds relative to the vowels.

Individual speaker influence on relative intelligibility of esophageal speech and artificial larynx speech.

Comparisons were made of the relative intelligibility of esophageal and artificial larynx speech. Care was taken to minimize the variable effects of individual speaker characteristics. For example, rather than relying solely on data comparisons between groups of different speakers, contrasts were also made between samples from subjects who could use both means of alaryngeal speech. Fifteen proficient male laryngectomized speakers were selected. Five used only esophageal speech, five used only artificial larynx speech, and five produced both forms of speech. Recordings were made while the subjects read 50 PB (Phonetically Balanced) words embedded in a carrier phrase and were transcribed by 30 naive listeners. Differences in intelligibility between esophageal and artificial larynx speech resulted when data comparisons were based on different speaker groups. However, differences were not evident when data from the same speakers were compared. The results were interpreted to highlight the possible influence of individual speaker characteristics, rather than differences between communication methods.