Voice Data Research Articles

AI Consent Futures: A Case Study on Voice Data Collection with Clinicians

As new forms of data capture emerge to power new AI applications, questions abound about the ethical implications of these data collection practices. In this paper, we present clinicians' perspectives on the prospective benefits and harms of voice data collection during health consultations. Such data collection is being proposed as a means to power models to assist clinicians with medical data entry, administrative tasks, and consultation analysis. Yet, clinicians' attitudes and concerns are largely absent from the AI narratives surrounding these use cases, and the academic literature investigating them. Our qualitative interview study used the concept of an informed consent process as a type of design fiction, to support elicitation of clinicians' perspectives on voice data collection and use associated with a fictional, near-term AI assistant. Through reflexive thematic analysis of in-depth sessions with physicians, we distilled eight classes of potential risks that clinicians are concerned about, including workflow disruptions, self-censorship, and errors that could impact patient eligibility for services. We conclude with an in-depth discussion of these prospective risks, reflect on the use of the speculative processes that illuminated them, and reconsider evaluation criteria for AI-assisted clinical documentation technologies in light of our findings.

Abnormal Articulation Detecting Model with Fluctuation Measurements Using Acoustic Analysis

Articulation disorder is a condition in which the mouth, tongue, vocal cords, and other parts of the body that play an important role in producing voice are damaged, resulting in the inability to produce sound. To diagnose articulation disorders, the movement and shape of each organ concerned with pronunciation are examined. If necessary, the underlying disease or disorder should be managed properly. In it, a speech therapist tests your pronunciation. The observation of conversation and the examination of the pronunciation of each syllable are used to distinguish between mistakes and the degree of articulation disorder. However, these processes are time consuming and labor intensive and are subjective judgments by experts. Therefore, it is important to investigate the characteristics of vocal signals by acoustic analysis of speech objectively. In this study, we focused on fluctuations in the period and amplitude of speech signals and predicted a model for detecting abnormal articulations using fluctuation measurement of the voice data in six healthy subjects and nine patients with an articulation disorder. We used inverse probability of treatment weighting to match the variability for the two groups using the inverse of propensity scores. As the results, the classification performance area under the curve was 0.781 (sensitivity: 0.781, specificity: 0.680) for healthy subjects and patients. We conclude that acoustic analyzing techniques are useful for diagnosing and treating articulation disorders.

Multi-Modality Tensor Fusion Based Human Fatigue Detection

Multimodal learning is an expanding research area and aims to pursue a better understanding of given data by regarding different modals. Multimodal approaches for qualitative data are used for the quantitative proofing of ground-truth datasets and discovering unexpected phenomena. In this paper, we investigate the effect of multimodal learning schemes of quantitative data to assess its qualitative state. We try to interpret human fatigue levels through analyzing video, thermal image and voice data together. The experiment showed that the multimodal approach using three types of data was more effective than the method of using each dataset individually. As a result, we identified the possibility of predicting human fatigue states.

Decision Making in Health Care Diagnosis: Evidence From Parkinson's Disease Via Hybrid Machine Learning

Health care is a complex system that demands critical decision making, especially in the diagnosis of various conditions in patients. To minimize possible errors in diagnosis, an emerging technology, machine learning (ML), is being effectively used. ML classifiers can be used to proactively diagnose the medical conditions, which are identified based on the presence or absence of specific characteristics of the diseases. Therefore, in this article, we demonstrate how ML can be used to determine Parkinson's disease (PD) and thereby, provide early diagnosis using nonclinical data of the patients. Novel ensembles are developed in this article to improve the diagnostic capability and the experimental results show that the improved versions of artificial neural network (ANN) could yield 13.4% more accurate results compared with the traditional ANN classifier. PD is considered a challenging medical condition, owing to its global relevance and complexity in diagnosis. Moreover, the early detection of PD is instrumental for patient recovery, and any lapses in diagnosis can lead to an immeasurable loss to patients. Also, the study has developed an effective diagnostic tool for PD and detects the disease at an early stage using the voice data of individuals, and this will aid in making better clinical decisions related to PD, thus rendering better health services.

SchröWave: Realistic voice generation by solving two-stage conditional Schrödinger bridge problems

In recent years, methods based on discrete diffusion models have achieved state-of-the-art performances in voice generation [1,2]. In theory, the voice data can be transformed into the exact Gaussian prior distributions only when the diffusion time tends to infinity. But in real applications, the Gaussian prior distribution can only be achieved approximately in a limited time duration run by these diffusion-based methods, thus resulting in sub-optimal sound quality. In this paper, we present the SchröWave to realize the continuous transformation from exact Dirac's deltas to the target voice data distribution in finite time duration, conditioned on middle voice representation with different sizes. At the same time, in order to overcome the difficulty in calculating the score on the low-dimensional manifold of voice data during the generation process, we propose to use a two-stage diffusion and generation method, while each stage implemented by solving a conditional Schrödinger bridge problem. Our experiments on the public data set LJSpeech show that the effect is significant in both objective and subjective evaluation, and achieve the new state-of-the-art MOS of 4.53. Audio samples are available at https://schrowave.github.io.

Clinical use of voice measurement in patients with heart failure: study protocol

Abstract Funding Acknowledgements Type of funding sources: None. Introduction Chronic conditions are the main causes of morbidity and mortality worldwide. Heart failure (HF) has a prevalence between 0.5% and 2% in the western population and the number is destined to increase from 25% in 2012 to 46% in 2030. Older adults affected by HF have difficulties to recognize signs and symptoms of the disease, bringing to a reduction of life expectancy. The voice is a complex coordination mechanism between systems and subsystems (i.e. glottis, lung, nose,..). Changes in voice parameters, measuring by algorisms of artificial intelligence, could be useful markers of HF exacerbation. To date, no prior studies have investigated the changing of voice parameters in HF patients and the relationships between voice parameter changes and the clinical course of HF. Aims Explore voice parameter differences between HF patients and healthy subjects; to investigate the relationships between voice parameter changes and the clinical course of the disease. Methods Observational study. Patients with HF and healthy subjects between 60 to 80 y/o will be enrolled in inpatient and outpatient settings. Patients and healthy participants will record voices by microphones and smartphones. Voice data will be analyzed with algorithms of artificial intelligence. Results This study will identify differences in voice parameters between HF patients and healthy participants, and the associations between the voice parameters and the clinical changes of HF patients. Conclusion Monitoring of voice parameters allows to clinicians to early identify, via voice changes, signs and symptoms of HF exacerbations, and to monitor patients at home.

Design and Implementation of a Small Program for Collecting Ethnic Voice

The Ethnic Language Collection Mini Program is an application based on mobile internet technology, designed to help users collect and record voice and text information of ethnic languages. The use of ethnic language collection mini programs can effectively promote the inheritance and protection of ethnic languages, helping users better understand and learn about ethnic culture. Language researchers can share their voice data with other users or research institutions to promote language research and protection.

PassivePy: A tool to automatically identify passive voice in big text data

AbstractThe academic study of grammatical voice (e.g., active and passive voice) has a long history in the social sciences. It has been examined in relation to psychological distance, attribution, credibility, and deception. Most evaluations of passive voice are experimental or small‐scale field studies, however, and perhaps one reason for its lack of adoption is the difficulty associated with obtaining valid, reliable, and replicable results through automated means. We introduce an automated tool to identify passive voice from large‐scale text data, PassivePy, a Python package (readymade website: https://passivepy.streamlit.app/). This package achieves 98% agreement with human‐coded data for grammatical voice as revealed in two large validation studies. In this paper, we discuss how PassivePy works, and present preliminary empirical evidence of how passive voice connects to various behavioral outcomes across three contexts relevant to consumer psychology: product complaints, online reviews, and charitable giving. Future research can build on this work and further explore the potential relevance of passive voice to consumer psychology and beyond.

Artificial Intelligence-Based Speech Analysis System for Medical Support.

Prior research has indicated that stroke can influence the symptoms and presentation of neurogenic bladder, with various patterns emerging, including abnormal facial and linguistic characteristics. Language patterns, in particular, can be easily recognized. In this paper, we propose a platform that accurately analyzes the voices of stroke patients with neurogenic bladder, enabling early detection and prevention of the condition. In this study, we developed an artificial intelligence-based speech analysis diagnostic system to assess the risk of stroke associated with neurogenic bladder disease in elderly individuals. The proposed method involves recording the voice of a stroke patient while they speak a specific sentence, analyzing it to extract unique feature data, and then offering a voice alarm service through a mobile application. The system processes and classifies abnormalities, and issues alarm events based on analyzed voice data. In order to assess the performance of the software, we first obtained the validation accuracy and training accuracy from the training data. Subsequently, we applied the analysis model by inputting both abnormal and normal data and tested the outcomes. The analysis model was evaluated by processing 30 abnormal data points and 30 normal data points in real time. The results demonstrated a high test accuracy of 98.7% for normal data and 99.6% for abnormal data. Patients with neurogenic bladder due to stroke experience long-term consequences, such as physical and cognitive impairments, even when they receive prompt medical attention and treatment. As chronic diseases become increasingly prevalent in our aging society, it is essential to investigate digital treatments for conditions like stroke that lead to significant sequelae. This artificial intelligence-based healthcare convergence medical device aims to provide patients with timely and safe medical care through mobile services, ultimately reducing national social costs.

We Need to be as a Group: Using and Evaluating the Listening Guide in Feminist Collaborative Autoethnography With an Affective ‘Fifth Listen’ as a Tool to (re)construct Identities

As a UK-based group of female postgraduate researchers, the authors explored their experiences during COVID-19 pandemic through multivocal inquiry via a feminist collaborative autoethnographic project. In this paper, we use the Listening Guide as a tool to revisit and (re)analyse data from the aforementioned project, displaying findings in the form of voice poems. In utilising the Listening Guide, we discovered that listening is less of an exercise and more an art form. While the structured approach of the LG helped to enhance our understandings of wider individual experiences of disability and womanhood, identities that all authors inhabit, we were surprised to find that despite our established mutual trust and superficially similar experiences, we were unable to find emotional resonance through data that wasn’t our own voice. We also found that the traditional stepped process of the LG that incorporates four listens to the data left our interpretations feeling flat. Through reflexivity and the novel collaborative approach we undertook in this analysis, we identified and implemented an augmentation of the Listening Guide process. In this paper, we propose an additional fifth listen, focusing on emotion, to facilitate a more holistic analysis of voice data. We explore how the fifth listen assisted the (re)construction of individual and collective identities, helping us to reshape our understandings. Finally, we elucidate the positives and pitfalls we experienced in the Listening Guide as a data analysis tool, recommending to other researchers the adoption of an iterative, flexible and reflexive approach in using it during collaborative research.

CONCEPTUAL REGIONAL ORIGIN RECOGNITION USING CNN CONVOUTION NEURAL NETWORK ON BANDUNG, BOGOR AND CIREBON REGIONAL ACCENTS

Sound detection is a challenge in machine learning due to the noisy nature of signals, and the small amount of (labeled) data that is usually available. The need for sound detection in Indonesia is quite important because there are many community organizations that form groups according to the land of their origin. Especially in big cities, where people from various tribes gather and exchange cultures. However, it has a disadvantage that affects these tribes, namely the loss of the original culture of certain areas. The Sundanese are the object of this research, including Bandung, Bogor and Cirebon. Voice data is divided into 2 types, namely male and female, each region consists of 50 respondents with 25 male and female voices with a maximum voting time of 1 minute. The method used is CNN architecture based on supervised learning, preprocessing using MFCC (Mel Frequency Cepstral Coefficients) to obtain feature extraction from voice data. CNN architecture is carried out 3 times convolution with max pooling and dropout on each convolution.

VoiceCloak

Faced with the threat of identity leakage during voice data publishing, users are engaged in a privacy-utility dilemma when enjoying the utility of voice services. Existing machine-centric studies employ direct modification or text-based re-synthesis to de-identify users' voices but cause inconsistent audibility for human participants in emerging online communication scenarios, such as virtual meetings. In this paper, we propose a human-centric voice de-identification system, VoiceCloak, which uses adversarial examples to balance the privacy and utility of voice services. Instead of typical additive examples inducing perceivable distortions, we design a novel convolutional adversarial example that modulates perturbations into real-world room impulse responses. Benefiting from this, VoiceCloak could preserve user identity from exposure by Automatic Speaker Identification (ASI), while remaining the voice perceptual quality for non-intrusive de-identification. Moreover, VoiceCloak learns a compact speaker distribution through a conditional variational auto-encoder to synthesize diverse targets on demand. Guided by these pseudo targets, VoiceCloak constructs adversarial examples in an input-specific manner, enabling any-to-any identity transformation for robust de-identification. Experimental results show that VoiceCloak could achieve over 92% and 84% successful de-identification on mainstream ASIs and commercial systems with excellent voiceprint consistency, speech integrity, and audio quality.

CovidCoughNet: A new method based on convolutional neural networks and deep feature extraction using pitch-shifting data augmentation for covid-19 detection from cough, breath, and voice signals

This study proposes a new deep learning-based method that demonstrates high performance in detecting Covid-19 disease from cough, breath, and voice signals. This impressive method, named CovidCoughNet, consists of a deep feature extraction network (InceptionFireNet) and a prediction network (DeepConvNet). The InceptionFireNet architecture, based on Inception and Fire modules, was designed to extract important feature maps. The DeepConvNet architecture, which is made up of convolutional neural network blocks, was developed to predict the feature vectors obtained from the InceptionFireNet architecture. The COUGHVID dataset containing cough data and the Coswara dataset containing cough, breath, and voice signals were used as the data sets. The pitch-shifting technique was used to data augmentation the signal data, which significantly contributed to improving performance. Additionally, Chroma features (CF), Root mean square energy (RMSE), Spectral centroid (SC), Spectral bandwidth (SB), Spectral rolloff (SR), Zero crossing rate (ZCR), and Mel frequency cepstral coefficients (MFCC) feature extraction techniques were used to extract important features from voice signals. Experimental studies have shown that using the pitch-shifting technique improved performance by around 3% compared to raw signals. When the proposed model was used with the COUGHVID dataset (Healthy, Covid-19, and Symptomatic), a high performance of 99.19% accuracy, 0.99 precision, 0.98 recall, 0.98 F1-Score, 97.77% specificity, and 98.44% AUC was achieved. Similarly, when the voice data in the Coswara dataset was used, higher performance was achieved compared to the cough and breath studies, with 99.63% accuracy, 100% precision, 0.99 recall, 0.99 F1-Score, 99.24% specificity, and 99.24% AUC. Moreover, when compared with current studies in the literature, the proposed model was observed to exhibit highly successful performance. The codes and details of the experimental studies can be accessed from the relevant Github page: (https://github.com/GaffariCelik/CovidCoughNet).

Estimation of habit-related information from male voice data using machine learning-based methods

According to a survey on the cause of death among Japanese people, lifestyle-related diseases (such as malignant neoplasms, cardiovascular diseases, and pneumonia) account for 55.8% of all deaths. Three habits, namely, drinking, smoking, and sleeping, are considered the most important factors associated with lifestyle-related diseases, but it is difficult to measure these habits autonomously and regularly. Here, we propose a machine learning-based approach for detecting these lifestyle habits using voice data. We used classifiers and probabilistic linear discriminant analysis based on acoustic features, such as mel-frequency cepstrum coefficients (MFCCs) and jitter, extracted from a speech dataset we developed, and an X-vector from a pre-trained ECAPA-TDNN model. For training models, we used several classifiers implemented in MATLAB 2021b, such as support vector machines, K-nearest neighbors (KNN), and ensemble methods with some feature-projection options. Our results show that a cubic KNN method using acoustic features performs well on the sleep habit classification, while X-vector-based models perform well on smoking and drinking habit classifications. These results suggest that X-vectors may help estimate factors directly affecting the vocal cords and vocal tracts of the users (e.g., due to smoking and drinking), while acoustic features may help classify chronotypes, which might be informative with respect to the individuals’ vocal cord and vocal tract ultrastructure.

Speech Recognition pada Kontrol Gerak Tangan Bionik berbasis Raspberry Pi dan Arduino

<p><em>The bionic hand is a broad bio-mechatronic device that incorporates various concepts involving biology and mechatronics. In this journal, the authors research finger movements in bionic hands using voice commands. In designing this bionic hand, the author made a mechanical design using a 3D printing machine. The hardware design used by the Raspberry Pi to process voice data is then transmitted to Arduino to carry out servo movement commands so that it can move bionic fingers. Software design uses the Python language using the Google text-to-speech API. From this research, data was generated with 100% accuracy. The movement of bionic fingers moves according to the commands spoken.</em></p>

Performance Improvement of Speech Emotion Recognition Systems by Combining 1D CNN and LSTM with Data Augmentation

In recent years, the increasing popularity of smart mobile devices has made the interaction between devices and users, particularly through voice interaction, more crucial. By enabling smart devices to better understand users’ emotional states through voice data, it becomes possible to provide more personalized services. This paper proposes a novel machine learning model for speech emotion recognition called CLDNN, which combines convolutional neural networks (CNN), long short-term memory neural networks (LSTM), and deep neural networks (DNN). To design a system that closely resembles the human auditory system in recognizing audio signals, this article uses the Mel-frequency cepstral coefficients (MFCCs) of audio data as the input of the machine learning model. First, the MFCCs of the voice signal are extracted as the input of the model. Local feature learning blocks (LFLBs) composed of one-dimensional CNNs are employed to calculate the feature values of the data. As audio signals are time-series data, the resulting feature values from LFLBs are then fed into the LSTM layer to enhance learning on the time-series level. Finally, fully connected layers are used for classification and prediction. The experimental evaluation of the proposed model utilizes three databases: RAVDESS, EMO-DB, and IEMOCAP. The results demonstrate that the LSTM model effectively models the features extracted from the 1D CNN due to the time-series characteristics of speech signals. Additionally, the data augmentation method applied in this paper proves beneficial in improving the recognition accuracy and stability of the systems for different databases. Furthermore, according to the experimental results, the proposed system achieves superior recognition rates compared to related research in speech emotion recognition.

Body exposure and vocal analysis: validation of fundamental frequency as a correlate of emotional arousal and valence.

Vocal analysis of fundamental frequency (f0) represents a suitable index to assess emotional activation. However, although f0 has often been used as an indicator of emotional arousal and different affective states, its psychometric properties are unclear. Specifically, there is uncertainty regarding the validity of the indices of f0mean and f0variabilitymeasures (f0dispersion, f0range, and f0SD) and whether higher or lower f0 indices are associated with higher arousal in stressful situations. The present study therefore aimed to validate f0 as a marker of vocally encoded emotional arousal, valence, and body-related distress during body exposure as a psychological stressor. N = 73 female participants first underwent a 3-min, non-activating neutral reference condition, followed by a 7-min activating body exposure condition. Participants completed questionnaires on affect (i.e., arousal, valence, body-related distress), and their voice data and heart rate (HR) were recorded continuously. Vocal analyses were performed using Praat, a program for extracting paralinguistic measures from spoken audio. The results revealed no effects for f0 and state body dissatisfaction or general affect. F0mean correlated positively with self-reported arousal and negatively with valence, but was not correlated with HRmean/maximum. No correlations with any measure were found for any f0variabililtymeasures. Given the promising findings regarding f0mean for arousal and valence and the inconclusive findings regarding f0 as a marker of general affect and body-related distress, it may be assumed that f0mean represents a valid global marker of emotional arousal and valence rather than of concrete body-related distress. In view of the present findings regarding the validity of f0, it may be suggested that f0mean, but not f0variabilitymeasures, can be used to assess emotional arousal and valence in addition to self-report measures, which is less intrusive than conventional psychophysiological measures.

Challenges Introducing a Novel Health Assessment System in Disaster-prone Japan: Community-Oriented Approach for Comprehensive Healthcare in Emergency Situations (COACHES)

Introduction:Collecting real-time individual health data of all disaster-affected populations is usually considered impossible. The University of Kochi's team and its partners conceived a novel health assessment system named "Community Oriented Approach for Comprehensive Healthcare in Emergency Situations (COACHES)." It collects individual health data anonymously and records such data in a cloud-based database. The system runs on any personal mobile device by scanning a personal identification code (QR code). It is expected that anyone on site with qualifications in healthcare will run this system as a volunteer to ensure data reliability. The COACHES app development is in process, and its prototype is currently available. This study aimed to assess how people react to the novel system and see the feasibility of installation in Japan.Method:Two focus group interviews (FGI) were conducted in a small coastal village in western Japan. The village anticipates severe damages with days of isolation once a large-scale earthquake followed by a tsunami hits the area. The first FGI was held with nine purposively recruited participants. The second FGI was held one month after and included seven of the first FGI participants with one of the absent participants providing a written response. FGI was for one-hour each, with discussion following the system demonstration. The voice data during the FGI were recorded and analyzed. The research was approved by the University of Kochi IRB.Results:Some showed intense interest in the system, whereas concerns such as the privacy violation for using personal devices, the availability of healthcare personnel, or a fear that anonymous data collection may delay identifying a person in need of assistance.Conclusion:Further studies are proposed, particularly in recruiting volunteers, data storage in case of technical damage, and how to make people with limited IT literacy comfortable using the new system.

Hybrid optimization enabled deep learning model for Parkinson's disease classification

ABSTRACT The analysis of Parkinson's disease (PD) is an inspiring task that necessitates the analysis of numerous motor and non-motor indications. During analysis, some abnormalities are considered as important symptoms to analyze the disease. Hence, this research introduced the proposed chronological smart sunflower optimization Algorithm (CSSFOA) for classifying the PD from voice data and voice signal samples. For voice signal, the input signals are pre-processed by the Gaussian filter, and then the significant features are extracted from it. The selection of optimal features is done by the chronological smart flower optimization Algorithm (CSFOA). The proposed CSFOA-based feature selection method considered the features by the Bray–Curtis distance. The PD classification is done by the ZF-Net which is trained by proposed CSSFOA to increase the performance of classification. The experimental result reveals that the proposed CSSFOA_ZF-Net algorithm got a better testing accuracy of 0.945, a sensitivity of 0.919, and a specificity of 0.957.

An ensemble nearest neighbor boosting technique for prediction of Parkinson’s disease

Parkinson’s disease (PD) is a complex condition that affects an individual’s motor features and intensifies over time. Symptoms vary between individuals and can go undetected at an early stage. Voice data has shown potential in predicting PD. In recent years, Machine Learning (ML) has made significant progress, with ensemble ML approaches proving to be effective in various applications. This research aims to detect PD at an early stage using ensemble techniques on the Parkinson’s speech dataset (PSV), which contains several sound recordings with 27 input features and one target feature. Feature importance was computed using Mean Decrease in Impurity (F-MDI), Feature Permutation (F-PER), and Pearson’s Correlation (F-CORR) techniques. Several supervised ML baseline models were compared with the proposed Nearest Neighbor Boosting (NNB) technique, which combines k-Nearest Neighbor (k-NN) and Gradient Boosting (GB). The NNB technique outperformed the baseline models for multiple performance metrics, including accuracy, recall, precision, F-score, the area under the curve (AUC), receiver operating characteristic (ROC), and confusion matrix (CM). The findings suggest that the designed ensemble method can be a promising method for PD detection.