Built-in Microphone Research Articles

DeepVehicleSense: An Energy-Efficient Transportation Mode Recognition Leveraging Staged Deep Learning Over Sound Samples

In this paper, we present a new transportation mode recognition system for smartphones called DeepVehicleSense, which is widely applicable to mobile context-aware services. DeepVehicleSense aims at achieving three performance objectives: high accuracy, low latency, and low power consumption at once by exploiting sound characteristics captured from the built-in microphone while being on candidate transportations. To attain high energy efficiency, DeepVehicleSense adopts hierarchical accelerometer-based triggers that minimize the activation of the microphone of smartphones. Further, to achieve high accuracy and low latency, DeepVehicleSense makes use of non-linear filters that can best extract the transportation sound samples. For recognition of five different transportation modes, we design a deep learning based sound classifier using a novel deep neural network architecture with multiple branches. Our staged inference technique can significantly reduce runtime and energy consumption while maintaining high accuracy for the majority of samples. Through 263-hour datasets collected by seven different Android phone models, we demonstrate that DeepVehicleSense achieves the recognition accuracy of 97.44\% with only sound samples of 2 seconds at the power consumption of 35.08 mW on average for all-day monitoring.

UroSound: A Smartwatch-Based Platform to Perform Non-Intrusive Sound-Based Uroflowmetry.

Leveraging consumer technology such as smartwatches to objectively and remotely assess people with voiding dysfunction could capture unique features for prompt diagnosis of a disease. This paper presents the UroSound, the first platform that performs non-intrusive sound-based uroflowmetry with a smartwatch. We study the feasibility of using a smartwatch to assess how well the urinary tract functions by processing the sound generated when the urine stream hits the water level in the toilet bowl, which can be modelled through the sound envelope. Signal-based features related to the sound envelope were extracted from a smartwatch's built-in microphone. The constructed model achieves a good correlation between acoustic and standard uroflowmetry in terms of the voiding shape and it can extract relevant voiding parameters. This indicates that accurate and remote measurement of the ambulatory characteristics of voiding dysfunction can be achieved with smartwatch-based uroflowmetry. UroSound also facilitates the collection of a voiding diary by measuring multiple uroflows during daytime and nighttime. Finally, the performance of 6 commercial smartwatches was analysed while recording a voiding event. The results demonstrate that the presence of an automatic gain control in the smartwatch microphone has a negative impact on the signal envelope, and should be avoided. Overall, this work demonstrates the potential for the use of smartwatches in the assessment of voiding dysfunction, to deliver more personalized and effective health care at home with less waste of time and resources, in particular in rural or less developed areas where access to a urology specialist is more difficult.

Vision Gloves for Blinds

For people who are visually impaired finding things in an indoor setting that are present in a typical environment might be a difficult chore in a visually impaired person's daily routine. They must be assisted by one or the other to find an object, so to overcome this, an idea has been proposed to develop a glove that can be used by visually impaired individuals to detect objects in an indoor environment and will also function as an artificial eye for those with visual impairments. The glove will have a camera in the hand that will feed real-time footage to the Raspberry Pi so that it can process it, a voice recognition module that will be attached to the Raspberry Pi to identify the object, and a micro vibrating motor to indicate the location of the object. The visually impaired person uses the speech module's built-in microphone to issue voice commands. An approach that uses object detection to find the object is used to locate it. By giving visually impaired people a hands-free means to find objects in enclosed places, the proposed smart glove has the potential to greatly improve their daily life. Hence an efficient and user-friendly system is planned to build by combining computer vision, voice recognition, and haptic feedback technology.

Intelligent gloves: An IT intervention for deaf-mute people

Abstract Deaf-mute people have much potential to contribute to society. However, communication between deaf-mutes and non-deaf-mutes is a problem that isolates deaf-mutes from society and prevents them from interacting with others. In this study, an information technology intervention, intelligent gloves (IG), a prototype of a two-way communication glove, was developed to facilitate communication between deaf-mutes and non-deaf-mutes. IG consists of a pair of gloves, flex sensors, an Arduino nano, a screen with a built-in microphone, a speaker, and an SD card module. To facilitate communication from the deaf-mutes to the non-deaf-mutes, the flex sensors sense the hand gestures and connected wires, and then transmit the hand movement signals to the Arduino nano where they are translated into words and sentences. The output is displayed on a small screen attached to the gloves, and it is also issued as voice from the speakers attached to the gloves. For communication from the non-deaf-mutes to the deaf-mute, the built-in microphone in the screen senses the voice, which is then transmitted to the Arduino nano to translate it to sentences and sign language, which are displayed on the screen using a 3D avatar. A unit testing of IG has shown that it performed as expected without errors. In addition, IG was tested on ten participants, and it has been shown to be both usable and accepted by the target users.

SymListener: Detecting Respiratory Symptoms via Acoustic Sensing in Driving Environments

Sound-related respiratory symptoms are commonly observed in our daily lives. They are closely related to illnesses, infections, or allergies but ignored by the majority. Existing detection methods either depend on specific devices, which are inconvenient to wear, or are sensitive to noises and only work for indoor environment. Considering the lack of monitoring method for in-car environment, where there is high risk of spreading infectious diseases, we propose a smartphone-based system, named SymListener, to detect respiratory symptoms in driving environment. By continuously recording acoustic data through a built-in microphone, SymListener can detect the sounds of cough, sneeze, and sniffle. We design a modified ABSE-based method to remove the strong and changeable driving noises while saving energy of the smartphone. An LSTM network is adopted to classify the three types of symptoms according to the carefully designed acoustic features. We implement SymListener on different Android devices and evaluate its performance in real driving environment. The evaluation results show that SymListener can reliably detect target respiratory symptoms with an average accuracy of 92.19% and an average precision of 90.91%.

SonicGest: Ultrasonic Gesture Recognition System Combined with GAN on Smartphones

In recent years, with the widespread popularity of mobile devices, gesture recognition as a way of human‐computer interaction has received more and more attention. However, existing gesture recognition methods have their limitations, such as requiring additional hardware devices, invading user privacy, and causing difficulty in data collection. To address these issues, we propose SonicGest, a recognition system that utilizes acoustic signals to sense in‐air gestures. The system only needs the built‐in speaker and microphone of the smartphone, without any additional hardware and no privacy disclosure. SonicGest transforms the features of the acoustic Doppler effect caused by gesture movements into a spectrogram, uses spectrogram enhancement techniques to remove noise interference, and then builds a convolutional neural network (CNN) classification model to recognize different gestures. To solve the problem of data collection difficulties, we utilize the Wasserstein distance based on gradient penalty to optimize the loss function of the generative adversarial network (GAN) to generate high‐quality spectrograms to expand the dataset. The experimental results show that SonicGest has a recognition accuracy of 98.9% for ten kinds of gestures.

Automated Bowel Sound and Motility Analysis with CNN Using a Smartphone

Bowel sound (BS) is receiving more attention as an indicator of gut health since it can be acquired non-invasively. Current gut health diagnostic tests require special devices that are limited to hospital settings. This study aimed to develop a prototype smartphone application that can record BS using built-in microphones and automatically analyze the sounds. Using smartphones, we collected BSs from 100 participants (age 37.6 ± 9.7). During screening and annotation, we obtained 5929 BS segments. Based on the annotated recordings, we developed and compared two BS recognition models: CNN and LSTM. Our CNN model could detect BSs with an accuracy of 88.9% andan F measure of 72.3% using cross evaluation, thus displaying better performance than the LSTM model (82.4% accuracy and 65.8% F measure using cross validation). Furthermore, the BS to sound interval, which indicates a bowel motility, predicted by the CNN model correlated to over 98% with manual labels. Using built-in smartphone microphones, we constructed a CNN model that can recognize BSs with moderate accuracy, thus providing a putative non-invasive tool for conveniently determining gut health and demonstrating the potential of automated BS research.

A Continuous Articulatory-Gesture-Based Liveness Detection for Voice Authentication on Smart Devices

Voice biometrics is drawing increasing attention to user authentication on smart devices. However, voice biometrics is vulnerable to replay attacks, where adversaries try to spoof voice authentication systems using prerecorded voice samples collected from genuine users. To this end, we propose VoiceGesture, a liveness detection solution for voice authentication on smart devices, such as smartphones and smart speakers. With audio hardware advances on smart devices, VoiceGesture leverages built-in speaker and microphone pairs on smart devices as Doppler radar to sense articulatory gestures for liveness detection during voice authentication. The experiments with 21 participants and different smart devices show that VoiceGesture achieves over 99% and around 98% detection accuracy for text-dependent and text-independent liveness detection, respectively. Moreover, VoiceGesture is robust to different device placements, low audio sampling frequency, and supports medium-range liveness detection on smart speakers in various use scenarios, including smart homes and smart vehicles.

Robust coupled tensor decomposition and feature extraction for multimodal medical data

High-dimensional and multimodal data to describe various aspects of a patient’s clinical condition have become increasingly abundant in the medical field across a variety of domains. For example, in neuroimaging applications, electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) can be collected simultaneously (i.e., EEG-fMRI) to provide high spatial and temporal resolution of a patient’s brain function. Additionally, in telemonitoring applications, a smartphone can be used to record various aspects of a patient’s condition using its built-in microphone, accelerometer, touch screen, etc. Coupled CANDECOMP/PARAFAC decomposition (CCPD) is a powerful approach to simultaneously extract common structures and features from multiple tensors and can be applied to these high-dimensional, multi-modal data. However, the existing CCPD models are inadequate to handle outliers, which are highly present in both applications. For EEG-fMRI, outliers are common due to fluctuations in the electromagnetic field resulting from interference between the EEG electrodes and the fMRI machine. For telemonitoring, outliers can result from patients not properly following instructions while performing smartphone-guided exercises at home. This motivates us to propose a robust CCPD (RCCPD) method for robust feature extraction. The proposed method utilizes the Alternating Direction Method of Multipliers (ADMM) to minimize an objective function that simultaneously decomposes a pair of coupled tensors and isolates outliers. We compare the proposed RCCPD method with the classical CP decomposition, the coupled matrix-tensor/tensor-tensor factorization (CMTF/CTTF), and the tensor robust CP decomposition (TRCPD). Experiments on both synthetic and real-world data demonstrate that the proposed RCCPD effectively handles outliers and outperforms the benchmarks in terms of accuracy.

A Low-Latency Streaming On-Device Automatic Speech Recognition System Using a CNN Acoustic Model on FPGA and a Language Model on Smartphone

This paper presents a low-latency streaming on-device automatic speech recognition system for inference. It consists of a hardware acoustic model implemented in a field-programmable gate array, coupled with a software language model running on a smartphone. The smartphone works as the master of the automatic speech recognition system and runs a three-gram language model on the acoustic model output to increase accuracy. The smartphone calculates and sends the Mel-spectrogram of an audio stream with 80 ms unit input from the built-in microphone of the smartphone to the field-programmable gate array every 80 ms. After ~35 ms, the field-programmable gate array sends the calculated word-piece probability to the smartphone, which runs the language model and generates the text output on the smartphone display. The worst-case latency from the audio-stream start time to the text output time was measured as 125.5 ms. The real-time factor is 0.57. The hardware acoustic model is derived from a time-depth-separable convolutional neural network model by reducing the number of weights from 115 M to 9.3 M to decrease the number of multiply-and-accumulate operations by two orders of magnitude. Additionally, the unit input length is reduced from 1000 ms to 80 ms, and to minimize the latency, no future data are used. The hardware acoustic model uses an instruction-based architecture that supports any sequence of convolutional neural network, residual network, layer normalization, and rectified linear unit operations. For the LibriSpeech test-clean dataset, the word error rate of the hardware acoustic model was 13.2% and for the language model, it was 9.1%. These numbers were degraded by 3.4% and 3.2% from the original convolutional neural network software model due to the reduced number of weights and the lowering of the floating-point precision from 32 to 16 bit. The automatic speech recognition system has been demonstrated successfully in real application scenarios.

Current State and Future Directions of Technologies for Music Instrument Pedagogy.

Technological advances over the past 50 years or so have resulted in the development of a succession of hardware and software systems intended to improve the quality and effectiveness of Western music instrument pedagogy during classroom instruction or individual study. These systems have aimed to provide evaluation or visualization of single or combined technical aspects by analyzing performance data collected in real time or offline. The number of such educational technologies shows an ever-increasing trend over time, aided by the wide diffusion and availability of mobile devices. However, we believe there are unrealized opportunities for modern technologies to help music students in their technical development and assist them during their practice sessions in between visits to their teachers. The ubiquity of PCs and mobile devices with built-in microphones, speakers, and cameras has inspired the development of media technologies in support of music pedagogy. They offer an attractive potential for implementing audio signal processing algorithms addressing different technical skills of the performer, providing real-time feedback, collecting data over time, and applying statistical models. Despite this potential, most available software for music instrument pedagogy remains very limited in functionality. This study provides a survey of music edTech software available, together with the methods of use, addressed technical skills, commonalities, and limitations. Results show that most current software is based on the metronome and tuner, with only a few systems that have limited abilities to follow a performance in real-time and compare it to a given score to monitor correctness of notes, intonation, and rhythm. The survey also highlights a high and under-exploited potential regarding the monitoring of other more specific technical skills, which are more instrument-dependent, but no less important, such as the control of dynamic range and clarity of the attack. This article ends with a discussion of possible directions for future development of technologies to support the practice of music students at different levels, with some consideration for the corresponding signal processing methods that can be utilized or that need advancement. By helping students to more efficiently achieve a high level of proficiency of their instruments with assistive technologies, we hope to minimize stress and afford better enjoyment of the music performance experience for all.

REPP: A robust cross-platform solution for online sensorimotor synchronization experiments

Sensorimotor synchronization (SMS), the rhythmic coordination of perception and action, is a fundamental human skill that supports many behaviors, including music and dance (Repp, 2005; Repp & Su, 2013). Traditionally, SMS experiments have been performed in the laboratory using finger tapping paradigms, and have required equipment with high temporal fidelity to capture the asynchronies between the time of the tap and the corresponding cue event. Thus, SMS is particularly challenging to study with online research, where variability in participants’ hardware and software can introduce uncontrolled latency and jitter into recordings. Here we present REPP (Rhythm ExPeriment Platform), a novel technology for measuring SMS in online experiments that can work efficiently using the built-in microphone and speakers of standard laptop computers. In a series of calibration and behavioral experiments, we demonstrate that REPP achieves high temporal accuracy (latency and jitter within 2 ms on average), high test-retest reliability both in the laboratory (r = .87) and online (r = .80), and high concurrent validity (r = .94). We also show that REPP is fully automated and customizable, enabling researchers to monitor experiments in real time and to implement a wide variety of SMS paradigms. We discuss online methods for ensuring high recruiting efficiency and data quality, including pre-screening tests and automatic procedures for quality monitoring. REPP can therefore open new avenues for research on SMS that would be nearly impossible in the laboratory, reducing experimental costs while massively increasing the reach, scalability, and speed of data collection.

ChestLive

Voice-based authentication is prevalent on smart devices to verify the legitimacy of users, but is vulnerable to replay attacks. In this paper, we propose to leverage the distinctive chest motions during speaking to establish a secure multi-factor authentication system, named ChestLive. Compared with other biometric-based authentication systems, ChestLive does not require users to remember any complicated information (e.g., hand gestures, doodles) and the working distance is much longer (30cm). We use acoustic sensing to monitor chest motions with a built-in speaker and microphone on smartphones. To obtain fine-grained chest motion signals during speaking for reliable user authentication, we derive Channel Energy (CE) of acoustic signals to capture the chest movement, and then remove the static and non-static interference from the aggregated CE signals. Representative features are extracted from the correlation between voice signal and corresponding chest motion signal. Unlike learning-based image or speech recognition models with millions of available training samples, our system needs to deal with a limited number of samples from legitimate users during enrollment. To address this problem, we resort to meta-learning, which initializes a general model with good generalization property that can be quickly fine-tuned to identify a new user. We implement ChestLive as an application and evaluate its performance in the wild with 61 volunteers using their smartphones. Experiment results show that ChestLive achieves an authentication accuracy of 98.31% and less than 2% of false accept rate against replay attacks and impersonation attacks. We also validate that ChestLive is robust to various factors, including training set size, distance, angle, posture, phone models, and environment noises.

Smartphone-based Handwritten Signature Verification using Acoustic Signals

Handwritten signature verification techniques, which can facilitate user authentication and enable secure information exchange, are still important in property safety. However, on-line automatic handwritten signature verification usually requires dynamic handwritten patterns captured by a special device, such as a sensor-instrumented pen, a tablet or a smartwatch on the dominant hand. This paper presents SonarSign, an on-line handwritten signature verification system based on inaudible acoustic signals. The key insight is to use acoustic signals to capture the dynamic handwritten signature patterns for verification. Particularly, SonarSign exploits the built-in speakers and microphones of smartphones to transmit a specially designed training sequence and record the corresponding echo for channel impulse response (CIR) estimation, respectively. Based on the sensitivity of CIR to the tiny surrounding environment changes including handwritten signature actions, SonarSign designs an attentional multi-modal Siamese network for end-to-end signatures verification. First, multi-modal CIR streams are fused to extract representative signature pattern features from spatio-temporal dimensions. Then an attentional Siamese network is elaborated to verify whether the given two signatures are from the same signatory. Extensive experiments in real-world scenarios show that SonarSign can achieve accurate and robust signatures verification with an AUC (Area Under ROC (Receiver Operating Characteristic) Curve) of 98.02% and an EER (Equal Error Rate) of 5.79% for unseen users.

Pandemic Robot

Abstract Robot-based service platforms are currently establishing themselves as new and affordable variants for supporting care in elderly, retirement and nursing homes. Many are open multifunctional platforms, which can potentially be integrated into such environments, if the necessary infrastructure is available. Furthermore, many services can be realized on these platforms, which can be used to foster distant interactions between inhabitants and care-providers, while simultaneously keeping up the quality of life of the inhabitants. Open mobile robotic platforms allow the extension with adequate new sensors. To detect infectious diseases of residents and healthcare-professionals, optical sensors can be used for the assessment of vital data such as heartrate and heartrate variability, respiratory rate, SpO2 or temperature. Additionally, you can consider demographic data (age, gender, constitution) of the observed person for the optical assessment, i.e. obtained by facial image analysis. As these mobile platforms are also equipped for telepresence, in case of detecting an infected person, these systems support video conferencing with their built-in cameras and microphones. Finally, the interaction with the electronic care record is necessary to upload all acquired vital data and further relevant information. All the named technologies have been under investigation in the past years and are currently moving from laboratory settings to real-world scenarios. Nevertheless, the smooth integration of all components into one system architecture in combination with (AI-based) data analysis are still open issues.

DriverSonar

Dangerous driving due to drowsiness and distraction is the main cause of traffic accidents, resulting in casualties and economic loss. There is an urgent need to address this problem by accurately detecting dangerous driving behaviors and generating real-time alerts. Inspired by the observation that dangerous driving actions induce unique acoustic features that respond to the signal of an acoustic source, we present the DriverSonar system in this paper. The proposed system detects dangerous driving actions and generates real-time alarms using off-the-shelf smartphones. Compared with the state-of-the-arts, the DriverSonar system does not require dedicated sensors but just uses the built-in speaker and microphone in a smartphone. Specifically, DriverSonar is able to recognize head/hand motions such as nodding, yawning, and abrupt adjustment of the steering wheel. We design, implement and evaluate DriverSonar with extensive experiments. We conduct both simulator-based and and real driving-based experiments (IRB-approved) with 30 volunteers for a period over 12 months. Experiment results show that the proposed system can detect drowsy and distraction related dangerous driving actions at an precision up to 93.2% and a low false acceptance rate of 3.6%.

Measurement of a spring force constant using sound, door alarm, and soundcard oscilloscope

The increasing or decreasing magnetic field of a permanent magnet—attached to the bottom of the mass hanger of a spring-mass oscillator—would turn ON or OFF, respectively, the switch of a door alarm system. The sound pulses—now voltage pulses as a result of conversion by the built-in microphone of a PC—were recorded by the PC-based oscilloscope as a function of time and on the basis of these data, the period of oscillations was obtained; thus, the spring constant. We obtained (29.6 ± 0.1) N m−1 which is a good match to the accepted value of 30.0 N m−1.

Sound frequency spectra of snore in relation to the site of obstruction among snorers.

SUMMARYObjectiveThis study aimed to describe the sound frequency of snoring in relation to the site of upper airway vibration among snorers.Methods383 snores from 40 participants who complained of snoring were digitally recorded during natural and induced sleep using a level III polysomnography monitor with a built-in microphone. During drug-induced sleep endoscopy (DISE), the real-time site of upper airway obstruction was assessed, and the sound frequency of snoring was recorded synchronously.ResultsThe mean peak of snoring frequency for unilevel palatal, oropharynx and epiglottis obstruction were 522.5, 482.4 and 300.0 Hz, respectively. Most participants showed multilevel obstruction at the palate and oropharynx, in which the mean for bi-peak snoring frequency were 402.90 Hz and 1086.96 Hz, respectively. Severity of OSA was significantly associated with multilevel obstruction.ConclusionsThere was a significant association between the snoring sound frequency and site of unilevel obstruction. Palatal or oropharyngeal obstruction produced sound at mid-frequency range, while the epiglottis produced a low frequency range. Multilevel obstruction documented a bi-peak snoring frequency.

Lung Auscultation Using the Smartphone-Feasibility Study in Real-World Clinical Practice.

Conventional lung auscultation is essential in the management of respiratory diseases. However, detecting adventitious sounds outside medical facilities remains challenging. We assessed the feasibility of lung auscultation using the smartphone built-in microphone in real-world clinical practice. We recruited 134 patients (median[interquartile range] 16[11–22.25]y; 54% male; 31% cystic fibrosis, 29% other respiratory diseases, 28% asthma; 12% no respiratory diseases) at the Pediatrics and Pulmonology departments of a tertiary hospital. First, clinicians performed conventional auscultation with analog stethoscopes at 4 locations (trachea, right anterior chest, right and left lung bases), and documented any adventitious sounds. Then, smartphone auscultation was recorded twice in the same four locations. The recordings (n = 1060) were classified by two annotators. Seventy-three percent of recordings had quality (obtained in 92% of the participants), with the quality proportion being higher at the trachea (82%) and in the children’s group (75%). Adventitious sounds were present in only 35% of the participants and 14% of the recordings, which may have contributed to the fair agreement between conventional and smartphone auscultation (85%; k = 0.35(95% CI 0.26–0.44)). Our results show that smartphone auscultation was feasible, but further investigation is required to improve its agreement with conventional auscultation.

Access Denied

Access Denied