Voice Information Research Articles

A survey of pilot behavior detection methods

This research investigates the use of action and gesture detection technology to analyze pilot behavior and ensure flight safety by employing artificial intelligence. In civil aviation, pilot and passenger safety is of paramount importance ,and the pilot behavior is an important factor in safety. Thus, it is necessary to detect the pilot behavior to secure the flight process which involves the detection of multiple modalities. In the past, most researchers have focused on single-modal behavioral detection methods, primarily gestures. However, a single modality, which may also include physiological data like EEG, is not a good and comprehensive understanding of a series of scenarios. Therefore, this paper proposes using multimodal data fusion to integrate behavior detection data from multiple sources. This paper summarizes and outlines the main methods of pilot behavior detection, such as cockpit voice, gesture detection and text information, and look forward to the possibility of multimodal data fusion of these there modalities data

Face and voice identity matching accuracy is not improved by multimodal identity information.

Identity verification from both faces and voices can be error-prone. Previous research has shown that faces and voices signal concordant information and cross-modal unfamiliar face-to-voice matching is possible, albeit often with low accuracy. In the current study, we ask whether performance on a face or voice identity matching task can be improved by using multimodal stimuli which add a second modality (voice or face). We find that overall accuracy is higher for face matching than for voice matching. However, contrary to predictions, presenting one unimodal and one multimodal stimulus within a matching task did not improve face or voice matching compared to presenting two unimodal stimuli. Additionally, we find that presenting two multimodal stimuli does not improve accuracy compared to presenting two unimodal face stimuli. Thus, multimodal information does not improve accuracy. However, intriguingly, we find that cross-modal face-voice matching accuracy predicts voice matching accuracy but not face matching accuracy. This suggests cross-modal information can nonetheless play a role in identity matching, and face and voice information combine to inform matching decisions. We discuss our findings in light of current models of person perception, and consider the implications for identity verification in security and forensic settings.

Development of a Voice Information Presentation Guidance System (VIPGS) Using Coanda‐Drone‐V2

AbstractIn this study, an aerial tour guide drone was developed based on the existing Coanda‐drone. The research integrated a Voice Information Presentation System with ROS2 using the open‐source PX4 flight control system to update the drone's control module. This system can recognize and read QR codes on displayed items and convert the text information from QR codes into speech for playback. It also utilizes left and right laser sensors for obstacle avoidance. Furthermore, the study conducted both simulation experiments and actual flight tests to validate the feasibility of equipping the Coanda‐drone with the voice information presentation system and to assess the flight capabilities of the Coanda‐drone. This system offers a novel solution for guidance and obstacle avoidance, demonstrating the potential of drones as flexible, highly secure, and efficient mobile service robots in the future. © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

Exploration of the Application of Multimodal Model in Psychological Analysis

Multimodal sentiment analysis is one of the important research areas in the field of artificial intelligence today. Multimodal sentiment analysis is to extract features from various human modalities such as facial expressions, body movements, and voice information, perform modal fusion, and finally classify and predict emotions. This technology can be used in multiple scenarios such as stock prediction, product analysis, movie box office prediction, etc., especially psychological state analysis, and has important research significance. This paper introduces two important datasets in multimodal sentiment analysis, namely CMU-MOSEI and IEMOCAP. It also introduces the feature-level fusion, model-level fusion, decision-level fusion and other fusion methods in multimodal fusion methods, and also introduces the semantic feature fusion neural network and sentiment word perception fusion network in multimodal sentiment analysis related models. Finally, the application of multimodal sentiment analysis models in depression and other related mental illnesses and the challenges of multimodal sentiment analysis models in the future are introduced. This paper hopes that the above research will be helpful for multimodal sentiment analysis.

Should voice navigation prompts be earlier and more complicated? A study of drivers’ visual attention, physiology and operating performance

Voice navigation systems can help drivers access information about road conditions and make correct decisions, but inappropriate information level and information prompt timing may lead to untimely processing of information, distraction, or even misjudgment of drivers, which in turn may cause safety hazards. The purpose of this study is to investigate the effects of voice information level and information prompt timing on drivers’ driving behavior and physiological characteristics. Twenty-seven participants were recruited to conduct driving simulator experiments. The results showed that the driver’s visual attention is more focused and the driving performance is better when the voice navigation prompts contain four information units (driving directions, traffic facility information, distance information, and road condition information at the same time). Moreover, when the voice navigation prompts with four information units were played 300 meters from the intersection, the driver’s overall driving performance was the highest. The results offer additional theoretical insights for the design of voice navigation systems, aiding drivers in accurately understanding driving conditions and providing them with ample reaction time to enhance their driving decisions.

Functional alterations of lateral temporal cortex for processing voice prosody in adults with autism spectrum disorder.

The human auditory system includes discrete cortical patches and selective regions for processing voice information, including emotional prosody. Although behavioral evidence indicates individuals with autism spectrum disorder (ASD) have difficulties in recognizing emotional prosody, it remains understudied whether and how localized voice patches (VPs) and other voice-sensitive regions are functionally altered in processing prosody. This fMRI study investigated neural responses to prosodic voices in 25 adult males with ASD and 33 controls using voices of anger, sadness, and happiness with varying degrees of emotion. We used a functional region-of-interest analysis with an independent voice localizer to identify multiple VPs from combined ASD and control data. We observed a general response reduction to prosodic voices in specific VPs of left posterior temporal VP (TVP) and right middle TVP. Reduced cortical responses in right middle TVP were consistently correlated with the severity of autistic symptoms for all examined emotional prosodies. Moreover, representation similarity analysis revealed the reduced effect of emotional intensity in multivoxel activation patterns in left anterior superior temporal cortex only for sad prosody. These results indicate reduced response magnitudes to voice prosodies in specific TVPs and altered emotion intensity-dependent multivoxel activation patterns in adult ASDs, potentially underlying their socio-communicative difficulties.

Voice interactive information metasurface system for simultaneous wireless information transmission and power transfer

Intelligent voice interaction offers a flexible and powerful way to connect individuals with smart devices beyond our expectations. The real-time nature of voice communication enables smart devices to comprehend the user language, execute the corresponding instructions, and facilitate seamless communications, transforming our lives in unprecedented ways. Owing to self-adaptive and reprogrammable functionalities, information metasurface (IMS) opens up a new avenue for smart home and smart cities. To further enhance the intelligence of IMS, we propose an IMS system via intelligent voice interaction and information processing. The voice interaction enables the efficient remote control on the IMS in a flexible, convenient, touchless manner. Leveraging speech recognition, speech synthesis, target detection, and communication technologies, the IMS system achieves automatic beam manipulation capabilities for wireless information transmissions and wireless power transfers. The IMS system is designed to operate in two distinct modes: instruction mode, wherein the user instructs the operations, and autonomous mode, wherein the automatic detections govern the actions, in which seamless mode switching through the voice commands is supported. Users can flexibly achieve precise control over the functions of the intelligent metasurface system through voice interaction at a distance, without the need for close-range manual touch control, which greatly simplifies the operation difficulty and is particularly suitable for remote control and complex application scenarios. A series of experiments, including wireless video transmissions and wireless power transfers are conducted to demonstrate the flexibility and convenience of the IMS system. The incorporation of intelligent voice interaction technology with the IMS presents a novel paradigm for the applications of programmable and information metasurfaces.

AR-NAVIS: Mobility Application for Blind and Deaf Students Based on Augmented Reality

Students with disabilities, especially those with visual and hearing impairments, face challenges in navigating through the campus environment. Hence, the development of AR-NAVIS as an Augmented Reality (AR)-based mobility orientation application stands as a significant innovation in providing services for them. This application aims to assist disabled students in moving within the campus environment, both indoors and outdoors. AR-NAVIS identifies the safest and most efficient routes, enabling disabled students to engage in independent activities and enhancing both their academic and non-academic performance. The application development process involves analyzing students' needs, prototype design, model validation, trials, and dissemination. Its features include AR-based 3D guidance, directional text, voice, vibration mode, and hazard information. The app is expected to provide accurate information about buildings or locations that are the destination for disabilities students. The result show that application development can guide disabilities user move between buildings smoothly. The experiment found that there was an increase in student activity after having this application.

Robust Dual-Modal Speech Keyword Spotting for XR Headsets.

While speech interaction finds widespread utility within the Extended Reality (XR) domain, conventional vocal speech keyword spotting systems continue to grapple with formidable challenges, including suboptimal performance in noisy environments, impracticality in situations requiring silence, and susceptibility to inadvertent activations when others speak nearby. These challenges, however, can potentially be surmounted through the cost-effective fusion of voice and lip movement information. Consequently, we propose a novel vocal-echoic dual-modal keyword spotting system designed for XR headsets. We devise two different modal fusion approches and conduct experiments to test the system's performance across diverse scenarios. The results show that our dual-modal system not only consistently outperforms its single-modal counterparts, demonstrating higher precision in both typical and noisy environments, but also excels in accurately identifying silent utterances. Furthermore, we have successfully applied the system in real-time demonstrations, achieving promising results. The code is available at https://github.com/caizhuojiang/VE-KWS.

Digital Measuring Instruments for Height and Weight with Microcontroller-Based Sound Output

Height and weight measuring devices simultaneously provide ideal weight information that will be handy for users. This study aims to provide an instrument that can be used as a measuring instrument for height and weight and whether or not a person's body is ideal. Therefore, in research, measuring instruments that can simultaneously measure height and weight and provide ideal information or not measured weight. This measuring instrument uses Arduino Uno as its brain, ultrasonic sensors to measure height, and strain gauge/load cell sensors to measure weight. Arduino processes data from both sensors to get the ideal body weight (BBI). The height, weight, and ideal body criteria values will be displayed on the LCD. Furthermore, the speaker will issue sound information regarding the condition of body weight, namely ideal, fat, or thin. Based on the testing and data analysis results, the average error percentage value in height measurement is 0.45%, and in weight measurement is 1.65%. At the same time, the success rate of the appearance of voice information is 99%.

Research on emotional analysis and integration of multimodal data report

In recent years, emotional analysis has remained a hot topic, permeating people's daily lives and playing an indispensable role in various environments. Researchers can improve products and services and forecast user behavior using sentiment analysis to understand consumers' emotional inclinations and reactions. This paper's comprehensive review of emotional analysis is conducted through various research data. This study integrates emotional analysis and multimodal data research, elaborating on emotional analysis's characteristics and pros and cons from aspects such as facial expressions, voice information, and textual data. This paper points out the current applications of emotional analysis and recent research advancements, selecting various features and methods for emotion tendency classification, and utilizing existing technological tools for analysis. Lastly, this work combines multimodal data with various studies to make emotional analysis more widely and aptly applied in everyday life. There are greater opportunities for sentiment analysis in various domains thanks to the advancements in multimodal sentiment analysis research and modal fusion technologies.

Individual investors’ online voice and corporate environmental information disclosure

This study explores the impact of individual investors’ online voice on corporate environmental information disclosure (EID). Using a sample of all Chinese A-share listed firms from 2009 to 2022, this study finds that individual investors’ online voice significantly improves corporate EID. This result remains robust through various robustness tests. It is further found that the positive effect of individual investors’ online voice on corporate EID is strengthened through external monitoring by institutional investors, analyst coverage, and media coverage. This positive effect is amplified in highly polluting industries. Additionally, individual investors' negative sentiment stimulates firms to disclose more environmental information.

Application of voice enhancement based on virtual sensor network in enterprise financial management

Based on virtual sensor network, this paper explores the application of voice enhancement technology in enterprise financial management. Virtual sensor network is a low cost, easy to expand and flexible network system based on simulation and simulation technology. In this network system, the virtual voice sensor node can collect and process the voice information inside the enterprise. Speech enhancement technology uses signal processing and machine learning algorithms to reduce noise, enhance and clarify speech signals, thereby improving the accuracy and reliability of speech data. By constructing virtual voice sensor nodes in virtual sensor networks and applying voice enhancement technology, the research results show that enterprises can collect and process voice data more accurately in the process of financial management. Voice enhancement technology can reduce the impact of environmental noise on financial data and improve the clarity of voice signals, so as to effectively improve the quality of data. Through the voice enhancement technology based on virtual sensor network, high-quality voice data acquisition and processing can be realized in the financial management of enterprises, improve the accuracy and effect of financial decision-making, and reduce the cost and complexity of enterprises.

Audio-Visual Kinship Verification: A New Dataset and a Unified Adaptive Adversarial Multimodal Learning Approach.

Facial kinship verification refers to automatically determining whether two people have a kin relation from their faces. It has become a popular research topic due to potential practical applications. Over the past decade, many efforts have been devoted to improving the verification performance from human faces only while lacking other biometric information, for example, speaking voice. In this article, to interpret and benefit from multiple modalities, we propose for the first time to combine human faces and voices to verify kinship, which we refer it as the audio-visual kinship verification study. We first establish a comprehensive audio-visual kinship dataset that consists of familial talking facial videos under various scenarios, called TALKIN-Family. Based on the dataset, we present the extensive evaluation of kinship verification from faces and voices. In particular, we propose a deep-learning-based fusion method, called unified adaptive adversarial multimodal learning (UAAML). It consists of the adversarial network and the attention module on the basis of unified multimodal features. Experiments show that audio (voice) information is complementary to facial features and useful for the kinship verification problem. Furthermore, the proposed fusion method outperforms baseline methods. In addition, we also evaluate the human verification ability on a subset of TALKIN-Family. It indicates that humans have higher accuracy when they have access to both faces and voices. The machine-learning methods could effectively and efficiently outperform the human ability. Finally, we include the future work and research opportunities with the TALKIN-Family dataset.

A digital teleneuropsychology platform for the diagnosis of mild cognitive impairment: from concept to certification as a medical device.

Innovative digital solutions are shaping a new concept of dementia care, opening additional venues for prevention, diagnosis, monitoring and treatment. Hereby, we report the development of a tablet-based teleneuropsychology platform (Tenèpsia®), from concept to certification as Medical Device (MD) Class IIA, as per new MD regulation 745/2017. The platform was designed for the remote cognitive evaluation and created thanks to the effort of a collaborative working group including experts from three Italian scientific societies and Biogen Italia S.r.l. (hereafter "Biogen"), and developers from Xenia Reply and Inside AI. The development strategy was guided by converting traditional paper-and-pencil tests into digital versions while maintaining comparable neuropsychological features and optimizing patient accessibility and user experience. The experts focused on the choice and adaptation of traditional neuropsychology measures for a 45-min teleneuropsychology assessment. The developers created a web and a mobile interface, respectively, for the professional (neuropsychologist) and non-professional (patient and caregiver) use. Recording of voice, drawing and typing information was enabled. Instant dashboards provide a quick overview of the patient's condition. Simulation activities were performed to obtain MD certification, valid across Europe. Neuropsychology services will benefit from the implementation in clinics of harmonized digital tools with adequate scientific and technological standards. The use of digital cognitive testing for the diagnosis of mild cognitive impairment is expected to enhance patient and clinician outcomes through simplified, digital objective data collection, sparing of time and resources, with a positive impact on healthcare costs and access to treatments, reducing inequalities and delays in diagnosis and cure.

A critical analysis of gender-based violence reporting and evidence building applications (GBVxTech) for capturing memory reports.

Gender-based violence (GBV) is under-reported to the authorities owing to the stigma, shame, and fear of reprisal that surrounds these crimes. To address this, there has been an influx of technologies, including mobile phone and online applications that allow victim-survivors (hereafter, victims) to document and report GBV (hereafter referred to as GBVxTech). We critically analysed the extent to which GBVxTech applications align with the scientific knowledge base on gathering accounts of crimes from victims and witnesses. We identified 41 reporting and evidence building applications from around the world but found many (n = 19) were no longer accessible. A total of 13 applications met the study criteria and were available for download. We evaluated each application on how well its design and features align with established minimum best practice standards for gathering complete and accurate accounts from witnesses and victims, such as the pre-interview instructions (e.g., setting ground rules), questioning approach (e.g., using open-ended questions), and the adequacy of security features (e.g., password protection). We found most applications employ open questions, encourage victims to report information in an independent voice, and seek to elicit information pertinent to a criminal investigation. None of the applications use leading questions. However, most applications do not establish ground rules, and many use forced-choice questions, do not time stamp the information gathered, or document when users change their answers. Many applications have limited security features, potentially compromising users' safety. Further, some applications do not provide information about how to use the app, an informed consent procedure, or data usage information. We discuss the findings and offer recommendations for future GBVxTech development.

Speech recognition enhancement based on wireless network sensors application in interactive intelligent teaching system

The speech function used in the multi-dimensional speech recognition model based on MTL and RNN is the commonly used MFCC function, and in feature extraction, after various filtering and conversion, deleting part of the speech information is a requirement for 3D audio recognition. Use multidimensional information as much as possible. Based on this, this article not only brings the development of convolutional neural network intelligent teaching, but also proposes a new education model. Heuristic education like flipped classroom is still the content of education research in recent years, and it is important to the current education form in China. It is mainly based on the traditional lecture mode or the classroom form. This article focuses on undergraduate education. It is designed in the classroom based on the video data of the undergraduate interactive activities and the interactive system. It has important research significance and practicality. At the same time, the interactive content of the traditional teaching method is used to perform voice recognition, and the recognition effect of the main body of the interactive system is further improved through the voice information. The spatial perspective is used to divide the lens perspective into seminars and experimental forms, and the pixel change value of each frame provides a specific reference for it to determine the reflection of student interaction activities and the evaluation of student seminars. This article introduces the implementation process of interactive content and interactive activities of the teaching system in the classroom, introduces the speech recognition framework of the MARS Chinese teaching system, uses the model principle, and compares the general speech data set test with the actual ARST model audio. The results show that, MAS model performance is more advantageous.

Analysis of the influence of selected audio pre-processing stages on accuracy of speaker language recognition

The subject matter of the study is the analysis of the influence of pre-processing stages of the audio on the accuracy of speaker language regognition. The importance of audio pre-processing has grown significantly in recent years due to its key role in a variety of applications such as data reduction, filtering, and denoising. Taking into account the growing demand for accuracy and efficiency of audio information classification methods, evaluation and comparison of different audio pre-processing methods becomes important part of determining optimal solutions. The goal of this study is to select the best sequence of stages of pre-processing audio data for use in further training of a neural network for various ways of converting signals into features, namely, spectrograms and mel-cepstral characteristic coefficients. In order to achieve the goal, the following tasks were solved: analysis of ways of transforming the signal into certain characteristics and analysis of mathematical models for performing an analysis of the audio series by selected characteristics were carried out. After that, a generalized model of real-time translation of the speaker's speech was developed and the experiment was planned depending on the selected stages of pre-processing of the audio. To conclude, the neural network was trained and tested for the planned experiments. The following methods were used: mel-cepstral characteristic coefficients, spectrogram, time mask, frequency mask, normalization. The following results were obtained: depending on the selected stages of pre-processing of voice information and various ways of converting the signal into certain features, it is possible to achieve speech recognition accuracy up to 93%. The practical significance of this work is to increase the accuracy of further transcription of audio information and translation of the formed text into the chosen language, including artificial laguages. Conclusions: In the course of the work, the best sequence of stages of pre-processing audio data was selected for use in further training of the neural network for different ways to convert signals into features. Mel-cepstral characteristic coefficients are better suited for solving our problem. Since the neural network strongly depends on its structure, the results may change with the increase in the volume of input data and the number of languages. But at this stage, it was decided to use only mel-cepstral characteristic coefficients with normalization.

Relationship between Smoking and Vocal Self-perception of Trans Women

Objective. To verify the relationship between smoking, age, schooling, and the vocal self-perception of trans women. Methods. Cross-sectional observational quantitative study conducted with 24 trans women over 18 years old, living in their affirmed gender for a minimum of 6 months. Data collection involved selected questions from the translated and authorized Portuguese version of the Trans Woman Voice Questionnaire (TWVQ) and information regarding age, education, occupation, and smoking status. All variables were analyzed descriptively, and the association with smoking was assessed using Pearson's Chi-square, Fisher's Exact, Student's T, and Mann-Whitney tests, with a significance level of 5%. Results. The mean age of trans women was 28.2 +/- 6.5 years (range: 21 - 48). Most participants (41.6%) had completed their high school education and pursued diverse careers. Regarding smoking habits, 58.3% of women were either current smokers or had smoked at least 100 cigarettes in their lifetime. A statistically significant association was found between smoking and age (p = 0.001), with smokers having a lower average age (24.9 years) compared to nonsmokers (32.9 years). However, no statistical significance was observed between smoking, education, and self-perception of vocal femininity. Only 9 (37.5%) trans women perceived their voices as feminine, while 17 (70.7%) desired a more feminine-sounding voice. Limitation. The limited sample size in this study may have constrained the ability to detect significant differences between the analyzed groups using statistical tests. Conclusion. The smoking prevalence was notably high among younger trans women. However, there was no statistically significant difference in vocal self-perception between those who smoked and those who did not. The majority of trans women expressed a desire for their voices to sound more feminine, particularly among smokers.

Understanding voice‐based information uncertainty: A case study of health information seeking with voice assistants

AbstractEvaluating information quality online is increasingly important for healthy decision‐making. People assess information quality using visual interfaces (e.g., computers, smartphones) with visual cues like aesthetics. Yet, voice interfaces lack critical visual cues for evaluating information because there is often no visual display. Without ways to assess voice‐based information quality, people may overly trust or misinterpret information which can be challenging in high‐risk or sensitive contexts. This paper investigates voice information uncertainty in one high‐risk context—health information seeking. We recruited 30 adults (ages 18–84) in the United States to participate in scenario‐based interviews about health topics. Our findings provide evidence of information uncertainty expectations with voice assistants, voice search preferences, and the audio cues they use to assess information quality. We contribute a nuanced discussion of how to inform more critical information ecosystems with voice technologies and propose ways to design audio cues to help people more quickly assess content quality.