Phone Classification Research Articles

Multilingual non-intrusive binaural intelligibility prediction based on phone classification

Speech intelligibility (SI) prediction models are a valuable tool for the development of speech processing algorithms for hearing aids or consumer electronics. For the use in realistic environments it is desirable that the SI model is non-intrusive (does not require separate input of original and degraded speech, transcripts or a-priori knowledge about the signals) and does a binaural processing of the audio signals. Most of the existing SI models do not fulfill all of these criteria. In this study, we propose an SI model based on phone probabilities obtained from a deep neural net. The model comprises a binaural enhancement stage for prediction of the speech recognition threshold (SRT) in realistic acoustic scenes. In the first part of the study, SRT predictions in different spatial configurations are compared to the results from normal-hearing listeners. On average, our approach produces lower errors and higher correlations compared to three intrusive baseline models. In the second part, we explore if measures relevant in spatial hearing, i.e., the intelligibility level difference (ILD) and the binaural ILD (BILD), can be predicted with our modeling approach. We also investigate if a language mismatch between training and testing the model plays a role when predicting ILD and BILD. This point is especially important for low-resource languages, where not thousands of hours of language material are available for training. Binaural benefits are predicted by our model with an error of 1.5 dB. This is slightly higher than the error with a competitive baseline MBSTOI (1.1 dB), but does not require separate input of original and degraded speech. We also find that good binaural predictions can be obtained with models that are not specifically trained with the target language.

Analysis of Gradient Boosting, XGBoost, and CatBoost on Mobile Phone Classification

In the ever-evolving landscape of mobile phone technology, accurately classifying device specifications is paramount for market analysis and consumer decision-making. This research conducts a comprehensive analysis of mobile phone specification classification using three prominent machine learning algorithms: Gradient Boosting, XGBoost, and CatBoost. Through meticulous dataset acquisition and preprocessing steps, including resolution normalization and price categorization, features essential for classification analysis were standardized. Robust cross-validation techniques were employed to assess model performance effectively. The study demonstrates the significant impact of normalization techniques on improving model performance across all algorithms and fold variations. CatBoost consistently emerges as the top-performing algorithm, followed closely by XGBoost, with Gradient Boosting displaying respectable performance. Notably, CatBoost consistently achieves the highest AUC values and accuracy scores, demonstrating superior performance in accurately classifying mobile phone specifications. These findings underscore the importance of preprocessing methods and algorithm selection in achieving optimal classification results. For mobile phone manufacturers, leveraging machine learning algorithms for effective classification can inform product development strategies, optimizing offerings based on consumer preferences. Similarly, for data analysts, employing appropriate preprocessing techniques and algorithmic approaches can lead to more accurate predictions and informed decision-making. Future research avenues include exploring advanced preprocessing methods, investigating alternative algorithms, and incorporating additional features or datasets to enrich the classification process. Overall, this research contributes to understanding mobile phone specification classification through machine learning methodologies, offering actionable insights for industry practitioners and researchers to address evolving market dynamics and consumer preferences.

What can phone attractors in RPS tell us? A study of dynamic information in speech signals for phone classification purposes

The speech production system is time-varying, multidimensional, and nonlinear. Most techniques for spoken feature extraction (SFE), which are tools for extracting information from speech signals, rely on the linear aspects of this system. In the past two decades, several techniques have been developed to account for the nonlinear characteristics of the system using embedded speech attractors in the reconstructed phase space (RPS). However, despite the clear benefits of speech representation in the RPS domain, only a few studies have successfully applied it for classification purposes. The main goal of this study is to develop an RPS-based framework that uses dynamic information of the embedded speech attractors in the RPS domain and outperforms the time-domain SFE techniques. The extracted features are based on multivariate linear prediction models of phone trajectories that show the dynamic information of the embedded speech attractor in the RPS. Several experiments on the FARSDAT and TIMIT databases test the phone classification accuracy of the proposed framework and show that the dynamic information of the phone attractors can significantly improve phone classification accuracy.

Frame-Based Phone Classification Using EMG Signals

This paper evaluates the impact of inter-speaker and inter-session variability on the development of a silent speech interface (SSI) based on electromyographic (EMG) signals from the facial muscles. The final goal of the SSI is to provide a communication tool for Spanish-speaking laryngectomees by generating audible speech from voiceless articulation. However, before moving on to such a complex task, a simpler phone classification task in different modalities regarding speaker and session dependency is performed for this study. These experiments consist of processing the recorded utterances into phone-labeled segments and predicting the phonetic labels using only features obtained from the EMG signals. We evaluate and compare the performance of each model considering the classification accuracy. Results show that the models are able to predict the phonetic label best when they are trained and tested using data from the same session. The accuracy drops drastically when the model is tested with data from a different session, although it improves when more data are added to the training data. Similarly, when the same model is tested on a session from a different speaker, the accuracy decreases. This suggests that using larger amounts of data could help to reduce the impact of inter-session variability, but more research is required to understand if this approach would suffice to account for inter-speaker variability as well.

Multilingual Non-intrusive Binaural Intelligibility Prediction based on Phone Classification

Multilingual Non-intrusive Binaural Intelligibility Prediction based on Phone Classification

Interpreting Deep Representations of Phonetic Features via Neuro-Based Concept Detector: Application to Speech Disorders Due to Head and Neck Cancer

The popularity of Deep Neural Networks (DNNs) is growing significantly, and so is the interest in gaining a better understanding of their functioning. In this work, it is even more interesting to reveal the behavior of these black-boxes since we are involved in a clinical context. To this end, we propose a general analytic framework, named <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Neuro-based Concept Detector (NCD)</i> , for interpreting deep representations of a DNN. Based on the activation patterns of the hidden neurons, this framework highlights the capacity of neurons to detect a specific concept related to the final task. The key strength of our framework is that it provides an interpretability tool for any type of DNN performing a classification task regardless of the application field. In this paper, we evaluate this framework on a Convolutional Neural Network (CNN) trained for the task of French phone classification. This choice was guided by the final objective of a long-term research project, which aims to identify the linguistic units best contributing to the maintenance or loss of intelligibility in the context of speech disorders. Through <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">NCD</i> , we demonstrate the emergence of phonetic features in the classification layers of the CNN-based model, while applied on healthy speech, a concept with a great interest in the field of clinical phonetics. Indeed, we further show that these interesting findings shed light on the characteristics of speech disorders in terms of altered phonetic features and provide relevant information for clinical practice, notably, patients' rehabilitation and follow-up.

Position information attention networks for explosive mobile phone classification

There is a growing concern about public security, especially the discovery of explosives hidden in mobile phones during security checks. However, there is almost no public dataset of explosive mobile phones to study this topic. We contribute the first explosive mobile phone benchmark dataset for security screening, named explosive mobile phones x-ray image dataset, which will be publicly available. Note that explosive mobile phone classification is a typical class imbalance task. Although the number of explosive mobile phones is far smaller than the number of normal mobile phones, one missed phone can cause a huge loss of life and property. To accurately identify explosives hidden in mobile phones, we propose a module called position information attention module (PIAM). Benefiting from aggregating the position information encoded in networks along the channel and spatial domains, PIAM highlights informative features of explosives. In addition, PIAM combines with other networks effectively at a low cost, empowering them with the ability to identify important details. Furthermore, in the face of the class imbalance, we propose a sample-oriented coefficient called sample cost with an update rule. Extensive experimental results show that PIAM and sample cost significantly improve the performance of many excellent networks in explosive mobile phone classification.

Domain Generalization for Language-Independent Automatic Speech Recognition.

A language-independent automatic speech recognizer (ASR) is one that can be used for phonetic transcription in languages other than the languages in which it was trained. Language-independent ASR is difficult to train, because different languages implement phones differently: even when phonemes in two different languages are written using the same symbols in the international phonetic alphabet, they are differentiated by different distributions of language-dependent redundant articulatory features. This article demonstrates that the goal of language-independence may be approximated in different ways, depending on the size of the training set, the presence vs. absence of familial relationships between the training and test languages, and the method used to implement phone recognition or classification. When the training set contains many languages, and when every language in the test set is related (shares the same language family with) a language in the training set, then language-independent ASR may be trained using an empirical risk minimization strategy (e.g., using connectionist temporal classification without extra regularizers). When the training set is limited to a small number of languages from one language family, however, and the test languages are not from the same language family, then the best performance is achieved by using domain-invariant representation learning strategies. Two different representation learning strategies are tested in this article: invariant risk minimization, and regret minimization. We find that invariant risk minimization is better at the task of phone token classification (given known segment boundary times), while regret minimization is better at the task of phone token recognition.

Unsupervised Speech Segmentation and Variable Rate Representation Learning Using Segmental Contrastive Predictive Coding

Typically, unsupervised segmentation of speech into the phone- and word-like units are treated as separate tasks and are often done via different methods which do not fully leverage the inter-dependence of the two tasks. Here, we unify them and propose a technique that can jointly perform both, showing that these two tasks indeed benefit from each other. Recent attempts employ self-supervised learning, such as contrastive predictive coding (CPC), where the next frame is predicted given past context. However, CPC only looks at the audio signal&#x2019;s frame-level structure. We overcome this limitation with a segmental contrastive predictive coding (SCPC) framework to model the signal structure at a higher level, e.g., phone level. A convolutional neural network learns frame-level representation from the raw waveform via noise-contrastive estimation (NCE). A differentiable boundary detector finds variable-length segments, which are then used to optimize a segment encoder via NCE to learn segment representations. The differentiable boundary detector allows us to train frame-level and segment-level encoders jointly. Experiments show that our single model outperforms existing phone and word segmentation methods on TIMIT and Buckeye datasets. We analyze the impact of the threshold on boundary detector performance, and our results suggest that automatically learning the boundary threshold can be as effective as manually tuning that threshold. We discover that phone class impacts the boundary detection performance, and the boundaries between successive vowels or semivowels are the most difficult. Finally, we use SCPC to extract speech features at the segment level rather than at the uniformly spaced frame level (e.g., 10 ms) and produce variable rate representations that change according to the contents of the utterance. We can lower the feature extraction rate from the typical 100 Hz to as low as 14.5 Hz on average while still outperforming the hand-crafted features such as MFCC on the linear phone classification task.

Multimodal Representations for Synchronized Speech and Real-Time MRI Video Processing

Representations for data subunits can help with recent data accumulation by enabling efficient storage and retrieval systems. In this paper, we investigate the problem of representation generation for phone classification and cross-modal same-different word discrimination tasks. The benefits of utilizing multimodal data on these tasks are examined together with different data fusion schemes. Mainly, the paper considers two different data modalities, upper airway mid-sagittal plane real-time magnetic resonance imaging (rtMRI) videos and the corresponding speech waveforms, and experiments on USC-TIMIT rtMRI dataset. For the phone classification task, two unimodal neural networks are designed, and these separate systems are merged in two different ways that provide data fusion between two modalities. The proposed networks differ in their stages in which they perform the data fusion. As hypothesized, our results show that data fusion indeed brings a performance improvement over both unimodal approaches, and performing fusion in earlier stages with cross-connections yields better results than fusing the data in later stages. In addition to the proposed phone classification schemes, different unimodal and multimodal systems are designed to obtain phone recognition results on USC-TIMIT rtMRI dataset. Phone representations generated for the phone classification task are also utilized in the phone recognition task, and their representative power is illustrated. Finally, we define a cross-view same-different word discrimination task on USC-TIMIT. We propose two different schemes to tackle this task, and find that for cross-view same-different discrimination, generating representations with the help of cross-modality yields better accuracy than a system employing independently created representations.

Learning Acoustic Word Embeddings With Dynamic Time Warping Triplet Networks

In the last years, acoustic word embeddings (AWEs) have gained significant interest in the research community. It applies specifically to the application of acoustic embeddings in the Query-by-Example Spoken Term Detection (QbE-STD) search and related word discrimination tasks. It has been shown that AWEs learned for the word or phone classification in one or several languages can outperform approaches that use dynamic time warping (DTW). In this paper, a new method of learning AWEs in the DTW framework is proposed. It employs a multitask triplet neural network to generate the AWEs. The triplet network learns acoustic representations of words through a comparison of DTW distances. In addition, a multitask objective, including a conventional word classification component, and a triplet loss component is proposed. The triplet loss component applies the DTW distance for the word discrimination task. The multitask objective ensures that the embeddings can be used with DTW directly. Experimental validation shows that the proposed approach is well-suited, but not necessarily restricted to the QbE-STD search. A comparison with several baseline methods shows that the new method leads to a significant improvement of the results on the word discrimination task. An evaluation of the word clustering in the learned embedding space is presented.

A flexible discriminative approach to automatic phone and broad phonetic group classification

In this work, we present a novel framework to phone and broad phonetic group (BPG) classification. The overall system adds discriminative power to the traditional HMM framework. All phones share one HMM. However, instead of using generative models (e.g., GMMs), our framework uses a discriminative classifier to predict the state probability (i.e., the probability of an HMM state given a feature vector input). Then, the optimal state sequence is decoded resulting in a time-alignment function between the acoustic feature vector sequence and the state sequence. For each state s, the corresponding feature vectors are averaged resulting in a single feature vector that represents the s-th vector of the block. Each phone class is represented by a block of feature vectors whose size is equal to the number of states. All feature vectors of the block are then concatenated to a single feature vector to represent a phone unit, which is used for a discriminative phone classifier. We validate our framework using the TIMIT database. The proposed framework with MFCCs has comparable performance to the-state-of-the-art phone classification algorithms, but with increased flexibility to account for duration and other features such as articulatory features. Improved performance for BPG classification is also observed.

Phone classification via manifold learning based dimensionality reduction algorithms

Mechanical limitations imposed on the articulators during speech production lead to a limitation of the intrinsic dimensionality of speech signals. This limitation leads to a specific neighborhood structure of speech sounds when they are represented in a high-dimensional feature space. We investigate whether phone classification can be improved by exploiting this neighborhood structure, by means of extended variants of the conventional Linear Discriminant Analysis (LDA) based on manifold learning.In this extended LDA approach, the within-class and between-class scatter matrices are defined in terms of adjacency graphs. We compare extensions of LDA that use either a full adjacency graph or an adjacency graph defined in the neighborhood of the training observations. In addition, we apply different kernels for weighing the distances in the graphs via different kernels, of which the Adaptive Kernel is proposed in this paper.Experiments with TIMIT show that while LDA algorithms that use the full adjacency graph do not outperform traditional LDA, the algorithms that exploit only local information provide significantly better results than traditional LDA. These improvements are not uniform across different broad phonetic classes, which suggests that the added value of the neighborhood structure is phone class dependent. The structure is represented by locally different densities in the neighborhood of feature vectors that are representative of a specific phone in a specific context.

Directly data-derived articulatory gesture-like representations retain discriminatory information about phone categories

How the speech production and perception systems evolved in humans still remains a mystery today. Previous research suggests that human auditory systems are able, and have possibly evolved, to preserve maximal information about the speaker's articulatory gestures. This paper attempts an initial step toward answering the complementary question of whether speakers’ articulatory mechanisms have also evolved to produce sounds that can be optimally discriminated by the listener's auditory system. To this end we explicitly model, using computational methods, the extent to which derived representations of “primitive movements” of speech articulation can be used to discriminate between broad phone categories. We extract interpretable spatio-temporal primitive movements as recurring patterns in a data matrix of human speech articulation, i.e., representing the trajectories of vocal tract articulators over time. To this end, we propose a weakly-supervised learning method that attempts to find a part-based representation of the data in terms of recurring basis trajectory units (or primitives) and their corresponding activations over time. For each phone interval, we then derive a feature representation that captures the co-occurrences between the activations of the various bases over different time-lags. We show that this feature, derived entirely from activations of these primitive movements, is able to achieve a greater discrimination relative to using conventional features on an interval-based phone classification task. We discuss the implications of these findings in furthering our understanding of speech signal representations and the links between speech production and perception systems.

Deep Scattering Spectrum

A scattering transform defines a locally translation invariant representation which is stable to time-warping deformation. It extends MFCC representations by computing modulation spectrum coefficients of multiple orders, through cascades of wavelet convolutions and modulus operators. Second-order scattering coefficients characterize transient phenomena such as attacks and amplitude modulation. A frequency transposition invariant representation is obtained by applying a scattering transform along log-frequency. State-the-of-art classification results are obtained for musical genre and phone classification on GTZAN and TIMIT databases, respectively.

Cell Phone Mobility Classification based on k-Clustering and Linear Classification

Road traffic data is an essential element of the intelligent transportation system. However, available traffic data is insufficient to meet the current needs due to the high investment of infrastructure and traffic sensors. Therefore, obtain the traffic data from cell phone information becomes an attractive option for its widely used, high opacity and low cost. In order to get the satisfying results, the large sample sizes are necessary for existing classification algorithms, which leads to the low real-time performance and high complexity. To overcome these problems, a mobility of cell phone classification method based on k- clustering and linear classification has been proposed in this paper. The experimental results demonstrate the higher stability and accuracy of proposed method.

Adjustment Method between Phonological Attributes and Phone Boundaries

Two kinds of imperfections, namely the detection errors and the asynchrony between phonological attributes and phone boundaries, can cause a substantial decline in recognition accuracy of a detection-based automatic speech recognition system. To solve these problems, an adjustment method between phonological attributes and phone boundaries is proposed in this paper. At first the prior knowledge of corpus and the detection results are combined, then the asynchronies in the phone boundary area are compensated and the detection errors are corrected; additionally, by selectively deleting some frames with errors, the precision of the phone models are improved. After adoption of this adjustment method, 1.4% of phoneme recognition rate can be improved in the TIMIT phone classification experiments based on Conditional Random Fields.

Tensor Deep Stacking Networks

A novel deep architecture, the tensor deep stacking network (T-DSN), is presented. The T-DSN consists of multiple, stacked blocks, where each block contains a bilinear mapping from two hidden layers to the output layer, using a weight tensor to incorporate higher order statistics of the hidden binary (½0; 1) features. A learning algorithm for the T-DSN’s weight matrices and tensors is developed and described in which the main parameter estimation burden is shifted to a convex subproblem with a closed-form solution. Using an efficient and scalable parallel implementation for CPU clusters, we train sets of T-DSNs in three popular tasks in increasing order of the data size: handwritten digit recognition using MNIST (60k), isolated state/phone classification and continuous phone recognition using TIMIT (1.1 m), and isolated phone classification using WSJ0 (5.2 m). Experimental results in all three tasks demonstrate the effectiveness of the T-DSN and the associated learning methods in a consistent manner. In particular, a sufficient depth of the T-DSN, a symmetry in the two hidden layers structure in each T-DSN block, our model parameter learning algorithm, and a softmax layer on top of T-DSN are shown to have all contributed to the low error rates observed in the experiments for all three tasks.

Modeling speech imitation and ecological learning of auditory-motor maps

Classical models of speech consider an antero-posterior distinction between perceptive and productive functions. However, the selective alteration of neural activity in speech motor centers, via transcranial magnetic stimulation, was shown to affect speech discrimination. On the automatic speech recognition (ASR) side, the recognition systems have classically relied solely on acoustic data, achieving rather good performance in optimal listening conditions. The main limitations of current ASR are mainly evident in the realistic use of such systems. These limitations can be partly reduced by using normalization strategies that minimize inter-speaker variability by either explicitly removing speakers' peculiarities or adapting different speakers to a reference model. In this paper we aim at modeling a motor-based imitation learning mechanism in ASR. We tested the utility of a speaker normalization strategy that uses motor representations of speech and compare it with strategies that ignore the motor domain. Specifically, we first trained a regressor through state-of-the-art machine learning techniques to build an auditory-motor mapping, in a sense mimicking a human learner that tries to reproduce utterances produced by other speakers. This auditory-motor mapping maps the speech acoustics of a speaker into the motor plans of a reference speaker. Since, during recognition, only speech acoustics are available, the mapping is necessary to “recover” motor information. Subsequently, in a phone classification task, we tested the system on either one of the speakers that was used during training or a new one. Results show that in both cases the motor-based speaker normalization strategy slightly but significantly outperforms all other strategies where only acoustics is taken into account.

Vowel Phoneme Recognition Based on Average Energy Information in the Zerocrossing Intervals and its Distribution Using ANN

Speech signal is modelled using the average energy of the signal in the zerocrossing intervals. Variation of these energies in the zerocrossing interval of the signal is studied and the distribution of this parameter through out the signal is evaluated. It is observed that the distribution patterns are similar for repeated utterances of the same vowels and varies from vowel to vowel. Credibility of the proposed parameter is verified over five Malayalam (one of the most popular Indian language) vowels using multilayer feed forward artificial neural network based recognition system. The performance of the system using additive white Gaussian noise corrupted speech is also studied for different SNR levels. From the experimental results it is evident that the average energy information in the zerocrossing intervals and its distributions can be effectively utilised for vowel phone classification and recognition.