Bone-conducted Speech Research Articles

Spectral analysis of bone-conducted speech using modified linear prediction

Spectral analysis of bone-conducted speech using modified linear prediction

A lightweight speech enhancement network fusing bone- and air-conducted speech.

Air-conducted (AC) microphones capture the high-quality desired speech and ambient noise, whereas bone-conducted (BC) microphones are immune to ambient noise but only capture band limited speech. This paper proposes a speech enhancement model that leverages the merits of BC and AC speech. The proposed model takes the spectrogram of BC and AC speech as input and fuses them by an attention-based feature fusion module. The backbone network of the proposed model uses the fused signals to estimate mask of the target speech, which is then applied to the noisy AC speech to recover the target speech. The proposed model adopts a lightweight design of densely gated convolutional attention network (DenGCAN) as the backbone network, which contains encoder, bottleneck layers, and decoder. Furthermore, this paper improves an attention gate and integrates it into skip-connections of DenGCAN, which allows the decoder to focus on the key areas of the feature map extracted by the encoder. As the DenGCAN adopts self-attention mechanism, the proposed model has the potential to improve noise reduction performance at the expense of an increased input-output latency. Experimental results demonstrate that the enhanced speech of the proposed model achieves an average 1.870 wideband-PESQ improvement over the noisy AC speech.

Online bone/air-conducted speech fusion in the presence of strong narrowband noise

Online adaptive bone/air-conducted speech fusion is a very promising and attractive technique for real time speech communication in noisy environments. This paper firstly demonstrates that conventional bone/air-conducted speech fusion systems may degrade significantly, despite that only a tiny narrowband noise pollution persists in bone-conducted speech by extensive simulations and experiments. To address this issue, a robust nonlinear system is proposed that successfully enhances the speech contaminated by the harsh narrowband noise. The penetrated narrowband noise is extracted by an adaptive linear prediction filter from an extra reference signal that is capture by a non-contacted bone sensor, and then fed to the parallel adaptive noise cancellers to reconstruct a cleaner error feedback for the speech recovery system. Due to this combination, the proposed system enjoys both attractive robustness against the harsh additive noise and strong real-time capability. Simulations as well as application to a real factory noise are conducted to demonstrate the superior performance of the proposed system.

Regularized Modified Covariance Method for Spectral Analysis of Bone-Conducted Speech

Regularized Modified Covariance Method for Spectral Analysis of Bone-Conducted Speech

Restoration of Bone-Conducted Speech With U-Net-Like Model and Energy Distance Loss

Restoration of Bone-Conducted Speech With U-Net-Like Model and Energy Distance Loss

Packet Loss Concealment Estimating Residual Errors of Forward-Backward Linear Prediction for Bone-Conducted Speech

Packet Loss Concealment Estimating Residual Errors of Forward-Backward Linear Prediction for Bone-Conducted Speech

A Two-Stage Approach to Quality Restoration of Bone-Conducted Speech

A Two-Stage Approach to Quality Restoration of Bone-Conducted Speech

Speech Signal Extraction Method Based on Bayesian Estimation Using Air- and Bone-Conduction Sound in Speech Confusion

With the recent development of digital information technology, speech information recognition system has improved dramatically and is being applied in a wide range of fields. In order to improve the system efficiency, countermeasures measures against noise from the surrounding environment have become an important issue. In the previous study, we derived an algorithm to estimate speech signals by using air-conducted speech signal mixed with noise as observation and bone-conducted speech signal. In this research, we propose an algorithm to obtain more accurate estimation results by using not only air-conducted speech signal mixed with noise but also bone-conducted speech signal as observation values. The effectiveness of the proposed method was confirmed experimentally by applying it to mixed speech signals actually measured in surrounding noisy environment.

Deep-Learning-Based Speech Emotion Recognition Using Synthetic Bone-Conducted Speech

Deep-Learning-Based Speech Emotion Recognition Using Synthetic Bone-Conducted Speech

Packet Loss Compensation for VoIP through Bone‐Conducted Speech Using Modified Linear Prediction

In this paper, we compare air‐conducted (AC) speech with bone‐conducted (BC) speech for the purpose of utilizing them in packet loss concealment (PLC) for speech quality of voice over internet protocol (VoIP). Instead of the autocorrelation method of linear prediction (LP), which was utilized in the conventional PLC techniques, we employ the modified covariance (MC) method. The MC method provides accurate LP estimation from short input data samples and avoids the numerical problem the autocorrelation method suffers from. The lost frame is compensated from both forward and backward directions in which linear gain and weighting are applied. When BC speech is used as the input speech data for PLC in the case where the speech sender is in noisy environments, BC speech behaves more accurately than the corresponding AC speech, resulting in an excellent performance of the LP‐based PLC technique. This unveils a useful use of BC speech in speech information systems. Experiments show that in severe noise environments of AC speech‐to‐noise ratio being less than 10 dB, BC speech is superior to AC speech for PLC. It is also shown that the transmitted BC speech is more accurately reconstructed for PLC than transmitted AC speech is done. © 2023 The Authors. IEEJ Transactions on Electrical and Electronic Engineering published by Institute of Electrical Engineer of Japan and Wiley Periodicals LLC.

Comparison of objective and subjective methods for evaluating speech quality and intelligibility recorded through bone conduction and in-ear microphones

Bone conduction and in-ear microphones pick up the Bone-conducted (BC) speech, which has traveled through bones and soft tissues. This BC speech is less sensitive to surrounding noise than Air-conducted (AC) speech. Therefore, there is an interest in recording this signal. However, the intelligibility and quality of these microphones are known to be a limit to their use. Previous work has noted confusion between vowel sounds in terms of intelligibility. To evaluate the quality and intelligibility, studies rely on subjective and objective tests. This paper aims to determine if the standard objective methods for rating speech quality and intelligibility can be applied to BC speech recorded through these microphones. In order to estimate intelligibility, a subjective test based on vowel recognition was compared to STOI and to a new criterion based on second formant frequencies of oral vowels. For speech quality estimation, MUSHRA and PESQ tests were compared. Results show difficulties in using objective methods instead of subjective ones. Therefore, for ongoing studies, it is suggested to use subjective methods to evaluate speech quality and intelligibility.

Methods for improving word intelligibility of bone-conducted speech by using bone-conduction headphones

Bone-conduction (BC) headphones enable listeners to hear sounds through BC while leaving the ear canal (EC) open to pass surrounding air-conducted (AC) sound at the same time. However, BC transmission degrades the word intelligibility of BC speech, especially in noisy environments. This article examines the attenuation of higher-frequency components during BC transmission as a key characteristic that degrades speech intelligibility and proposes two methods for improving the intelligibility of BC speech by means of BC headphones. We focused on two types of BC transfer function, namely vibration of the regio temporalis (RT) and sound radiated in the EC relative to AC speech sound, and developed two methods for each (first order high-frequency emphasis (FOE) and higher-order high-frequency emphasis (HOE)) for improving the intelligibility of BC speech by compensating for the BC transmission characteristics. Speech signals in the proposed methods are filtered using four types of emphasis filter (RT-FOE, EC-FOE, RT-HOE, and EC-HOE). The results of word-intelligibility tests demonstrated that the word recognition rate, especially with FOE by the inverse transfer function between RT vibration and AC speech (RT-FOE), tended to significantly increase compared to that with no emphasis. Our findings suggested that linear compensation for the attenuation of higher frequency due to BC transmission by using the inverse transfer function between RT vibration and AC speech, is effective for improving BC speech intelligibility.

Speaker-Independent Spectral Enhancement for Bone-Conducted Speech

Because of the acoustic characteristics of bone-conducted (BC) speech, BC speech can be enhanced to better communicate in a complex environment with high noise. Existing BC speech enhancement models have weak spectral recovery capability for the high-frequency part of BC speech and have poor enhancement and robustness for the speaker-independent BC speech datasets. To improve the enhancement effect of BC speech for speaker-independent speech enhancement, we use a GANs method to establish the feature mapping between BC and air-conducted (AC) speech to recover the missing components of BC speech. In addition, the method adds the training of the spectral distance constraint model and, finally, uses the enhanced model completed by the training to reconstruct the BC speech. The experimental results show that this method is superior to the comparison methods such as CycleGAN, BLSTM, GMM, and StarGAN in terms of speaker-independent BC speech enhancement and can obtain higher subjective and objective evaluation results of enhanced BC speech.

End-to-End Multi-Modal Speech Recognition on an Air and Bone Conducted Speech Corpus

Automatic speech recognition (ASR) has been significantly improved in the past years. However, most robust ASR systems are based on air-conducted (AC) speech, and their performances in low signal-to-noise-ratio (SNR) conditions are not satisfactory. Bone-conducted (BC) speech is intrinsically insensitive to environmental noise, and therefore can be used as an auxiliary source for improving the performance of an ASR at low SNR. In this paper, we first develop a multi-modal Mandarin corpus, which contains air- and bone-conducted synchronized speech (ABCS). The multi-modal speeches are recorded with a headset equipped with both AC and BC microphones. To our knowledge, it is by far the largest corpus for conducting bone conduction ASR research. Then, we propose a multi-modal conformer ASR system based on a novel multi-modal transducer (MMT). The proposed system extracts semantic embeddings from the AC and BC speech signals by a conformer-based encoder and a transformer-based truncated decoder. The semantic embeddings of the two speech sources are fused dynamically with adaptive weights by the MMT module. Experimental results demonstrate the proposed multi-modal system outperforms single-modal systems with either AC or BC modality and multi-modal baseline system by a large margin at various SNR levels. It also shows the two modalities complement with each other, and our method can effectively utilize the complementary information of different sources.

Multi-modal speech enhancement with bone-conducted speech in time domain

Bone-conducted (BC) speech captures speech signals based on the vibrations of a speaker’s skull. It is thus not affected by noise sources from environments and hence exhibits better noise-resistance capabilities than air-conducted (AC) speech. Although the quality and intelligibility of the BC speech degrade due to the nature of the solid vibration, BC speech can be utilized as an auxiliary source to jointly improve the performance of speech enhancement. In this paper, we propose an end-to-end multi-modal model for time-domain speech enhancement at low signal-to-noise ratios. The model utilizes both noisy AC speech and synchronized BC speech as the input. It takes an encoder-decoder architecture, where an involution network is used to estimate the mask of clean speech component, and the mask is then applied to remove the noise component. We compared the proposed method with several state-of-the-art multi-modal and single-modal methods on an air- and bone-conducted multi-modal corpus. Experimental results demonstrate that the proposed approach outperforms the comparison methods in terms of the speech quality and intelligibility of the enhanced speech. When applied to speech recognition, the enhanced speech significantly reduces the error rate.

Regional Language Speech Recognition from Bone-Conducted Speech Signals through Different Deep Learning Architectures.

Bone-conducted microphone (BCM) senses vibrations from bones in the skull during speech to electrical audio signal. When transmitting speech signals, bone-conduction microphones (BCMs) capture speech signals based on the vibrations of the speaker's skull and have better noise-resistance capabilities than standard air-conduction microphones (ACMs). BCMs have a different frequency response than ACMs because they only capture the low-frequency portion of speech signals. When we replace an ACM with a BCM, we may get satisfactory noise suppression results, but the speech quality and intelligibility may suffer due to the nature of the solid vibration. Mismatched BCM and ACM characteristics can also have an impact on ASR performance, and it is impossible to recreate a new ASR system using voice data from BCMs. The speech intelligibility of a BCM-conducted speech signal is determined by the location of the bone used to acquire the signal and accurately model phonemes of words. Deep learning techniques such as neural network have traditionally been used for speech recognition. However, neural networks have a high computational cost and are unable to model phonemes in signals. In this paper, the intelligibility of BCM signal speech was evaluated for different bone locations, namely the right ramus, larynx, and right mastoid. Listener and deep learning architectures such as CapsuleNet, UNet, and S-Net were used to acquire the BCM signal for Tamil words and evaluate speech intelligibility. As validated by the listener and deep learning architectures, the Larynx bone location improves speech intelligibility.

Magnetically Levitated Flexible Vibration Sensors with Surficial Micropyramid Arrays for Magnetism Enhancement.

Magnetically levitated vibration sensors possess wide frequency response ranges and high sensitivity. Compared with springs and cantilevers, the levitated magnet suffers no mechanical abrasion, allowing minimized mechanical fatigue after prolonged exposure to vibration. However, magnetic levitated sensors are mostly based on fully rigid components, which are difficult to match the soft and curvilinear surface of the biological tissues and machines. Here, an innovative vibration sensor based on magnetic levitation has been developed. The proposed sensor contains two parallel magnetic membranes, one of which is levitated by magnetic force and connected to a specially designed sensor package. The surfaces of the membranes are modified with micropyramid arrays to enhance the magnetism and integrated with flexible coil arrays to maximize the changes in magnetic flux during vibration. The sensor exhibits a wide frequency response ranging from 1 Hz to 20 kHz and high sensitivity of 0.82 mV/μm at an operating frequency of 120 Hz. Various applications have been demonstrated through bone-conducted speech acquisition, sound recording, human motion detection, and machine condition evaluation. The sensor is one of the first flexible vibration sensors based on magnetic levitation. Its innovative levitated sensing structures may inspire development of novel flexible sensors with soft mechanical moving structures for force and displacement sensing in healthcare and industrial monitoring.

Air‐Conducted and Bone‐Conducted Speeches Combination for Noise‐Robust Pitch Extraction

In this paper, we present a noise‐robust pitch extraction method in which air‐conducted (AC) speech and bone‐conducted (BC) speech are utilized simultaneously as the input signals. Due to noise independency in both the input signals and noise suppression effect in BC speech, peak characteristics created in different functions are significantly enhanced so that accurate pitch extraction is achieved even in highly noisy environments. Experimental results show a superior performance of the proposed method relative to the state‐of‐the art method in several types of noises. © 2022 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

1Pb5-7 Intelligibility of bone-conducted speech detected on the scalp

1Pb5-7 Intelligibility of bone-conducted speech detected on the scalp

Fusing Bone-conduction and Air-conduction Sensors for Complex-Domain Speech Enhancement.

Speech enhancement aims to improve the listening quality and intelligibility of noisy speech in adverse environments. It proves to be challenging to perform speech enhancement in very low signal-to-noise ratio (SNR) conditions. Conventional speech enhancement utilizes air-conduction (AC) microphones, which are sensitive to background noise but capable of capturing full-band signals. On the other hand, bone-conduction (BC) sensors are unaffected by acoustic noise, but recorded speech has limited bandwidth. This study proposes an attention-based fusion method to combine the strengths of AC and BC signals and perform complex spectral mapping for speech enhancement. Experiments on the EMSB dataset demonstrate that the proposed approach effectively leverages the advantages of AC and BC sensors, and outperforms a recent time-domain baseline in all conditions. We also show that the sensor fusion method is superior to single-sensor counterparts, especially in low SNR conditions. As the amount of BC data is very limited, we additionally propose a semi-supervised technique to utilize both parallelly and unparallely recorded AC and BC speech signals. With additional AC speech from the AISHELL-1 dataset, we achieve similar performance to supervised learning with only 50% parallel data.