Time-frequency Bin Research Articles

An iteratively reweighted steered response power approach to multisource localization using a distributed microphone network

The steered response power (SRP) with phase transform algorithm has been demonstrated to be robust against reverberation and noise for single-source localization. However, when this algorithm is applied to multisource localization (MSL), the “peak missing problem” can occur, namely, that some sources dominate over others over short time intervals, resulting in fewer significant SRP peaks being found than the true number of sources. This problem makes it difficult to detect all the sources among the available SRP peaks. We propose an iteratively reweighted steered response power (IR-SRP) approach that effectively solves the “peak missing problem” and achieves robust MSL in reverberant noisy environments. The initial IR-SRP localization function is computed over the time-frequency (T-F) bins selected by a combination of two weighting schemes, one using coherence, and the other using signal-to-noise ratio. When iterating, our method finds the significant SRP peaks for the dominant sources and eliminates the T-F bins contributed by these sources using inter-channel phase difference information. As a result, the remaining sources can be found in subsequent iterations among the remaining T-F bins. The proposed IR-SRP method is demonstrated using both simulated and measured experiment data.

Time-Frequency Bins Selection for Direction of Arrival Estimation Based on Speech Presence Probability Learning

Time-Frequency Bins Selection for Direction of Arrival Estimation Based on Speech Presence Probability Learning

CompNet: Complementary network for single-channel speech enhancement

Recent multi-domain processing methods have demonstrated promising performance for monaural speech enhancement tasks. However, few of them explain why they behave better over single-domain approaches. As an attempt to fill this gap, this paper presents a complementary single-channel speech enhancement network (CompNet) that demonstrates promising denoising capabilities and provides a unique perspective to understand the improvements introduced by multi-domain processing. Specifically, the noisy speech is initially enhanced through a time-domain network. However, despite the waveform can be feasibly recovered, the distribution of the time–frequency bins may still be partly different from the target spectrum when we reconsider the problem in the frequency domain. To solve this problem, we design a dedicated dual-path network as a post-processing module to independently filter the magnitude and refine the phase. This further drives the estimated spectrum to closely approximate the target spectrum in the time–frequency domain. We conduct extensive experiments with the WSJ0-SI84 and VoiceBank + Demand datasets. Objective test results show that the performance of the proposed system is highly competitive with existing systems.

Learning-based robust speaker counting and separation with the aid of spatial coherence

A three-stage approach is proposed for speaker counting and speech separation in noisy and reverberant environments. In the spatial feature extraction, a spatial coherence matrix (SCM) is computed using whitened relative transfer functions (wRTFs) across time frames. The global activity functions of each speaker are estimated from a simplex constructed using the eigenvectors of the SCM, while the local coherence functions are computed from the coherence between the wRTFs of a time-frequency bin and the global activity function-weighted RTF of the target speaker. In speaker counting, we use the eigenvalues of the SCM and the maximum similarity of the interframe global activity distributions between two speakers as the input features to the speaker counting network (SCnet). In speaker separation, a global and local activity-driven network (GLADnet) is used to extract each independent speaker signal, which is particularly useful for highly overlapping speech signals. Experimental results obtained from the real meeting recordings show that the proposed system achieves superior speaker counting and speaker separation performance compared to previous publications without the prior knowledge of the array configurations.

A frequency-dependent acoustic diversity index: A revision to a classic acoustic index for soundscape ecological research

In recent years, passive acoustic monitoring (PAM) has been widely applied in the field of soundscape ecology. Meanwhile, acoustic indices, as an important tool for rapid biodiversity assessment (RBA), have also attracted increasing attention. However, considerable evidence demonstrates that existing acoustic indices are susceptible to noise, which poses a challenge for wider applications of acoustic indices. In this work, we firstly provide insights into how noise affects the widely used acoustic diversity index (ADI), revealing that the global threshold scheme (i.e., −50 dBFS for all time–frequency bins) to generate the binary spectrogram (spectrogram containing time–frequency bins denoted as ‘1′ with power above the threshold and ‘0′ below) is the key factor that makes ADI sensitive to noise. To address this problem, we propose a revised ADI version named frequency-dependent acoustic diversity index (FADI), which employs a new threshold scheme to reduce noise impacts on its values. Specifically, considering that the noise floor along frequency axis usually varies in practice, i.e., a colored noise rather than a white noise exists in field recordings, floating thresholds adapted to the noise level at each frequency bin were employed and further incorporated with the global threshold of ADI. Controlled computational experiments using field recordings were conducted where manipulated noise levels were introduced as different signal-to-noise ratios (SNR). It was shown that the proposed FADI was significantly robust to noise within a much wider SNR range as compared with ADI. Furthermore, a singular example using a real-world recording also verified its effectiveness and highlighted the promising potential of FADI in applications of PAM.

Collaboration With Cellular Networks for RFI Cancellation at Radio Telescope

The growing need for electromagnetic spectrum to support the next generation (xG) communication networks increasingly generate unwanted radio frequency interference (RFI) in protected bands for radio astronomy. RFI is commonly mitigated at the Radio Telescope without any active collaboration with the interfering sources. In this work, we provide a method of signal characterization and its use in subsequent cancellation, that uses Eigenspaces derived from the telescope and the transmitter signals. This is different from conventional time-frequency domain analysis, which is limited to fixed characterizations (e.g., complex exponential in Fourier methods) that cannot adapt to the changing statistics (e.g., autocorrelation) of the RFI, typically observed in communication systems. We have presented effectiveness of this collaborative method using realworld astronomical signals and practical simulated LTE signals (downlink and uplink) as source of RFI along with propagation conditions based on preset benchmarks and standards. Through our analysis and simulation using these signals, we are able to remove 89.04% of the RFI from cellular networks, which reduces excision at the Telescope and is capable of significantly improving throughput as corrupted time-frequency bins of data become usable.

Inter-Frequency Phase Difference for Phase Reconstruction Using Deep Neural Networks and Maximum Likelihood

This paper presents improvements to two-stage algorithms for estimating the short-time Fourier transform (STFT) phase from only the amplitude by using deep neural networks (DNNs). The phase is difficult to reconstruct due to its sensitivity to the waveform shift and wrapping issue. To mitigate these problems, two-stage approaches indirectly estimate the phase through phase derivatives, i.e., instantaneous frequency (IF) and group delay (GD). In the first stage, the IF and GD are estimated from the amplitude using DNNs, and then in the second stage, the phase is reconstructed by maintaining the IF/GD information. Conventional methods for the second stage do not consider the importance of high-amplitude time–frequency bins, e.g., the least squares-based method, or lack a solid model, e.g., the average-based method. To address these problems, we propose improvements to the second stage of two-stage algorithms by using <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">von Mises</i> distribution-based maximum likelihood and weighted least squares. We also provide theoretical discussions for the phase reconstruction, including the investigations of the properties of the GD and roles of the IF/GD information in the inverse STFT. On the basis of the analysis, we propose a new phase-based feature, i.e., inter-frequency phase difference (IFPD), and demonstrate its application in two-stage phase reconstruction algorithms. We conducted subjective and objective experiments to compare the performances of our proposed and conventional methods. The results confirm that the proposed method using the IFPD performs better than other methods for all metrics.

Online Phase Reconstruction via DNN-Based Phase Differences Estimation

This paper presents a two-stage online phase reconstruction framework using causal deep neural networks (DNNs). Phase reconstruction is a task of recovering phase of the short-time Fourier transform (STFT) coefficients only from the corresponding magnitude. However, phase is sensitive to waveform shifts and not easy to estimate from the magnitude even with a DNN. To overcome this problem, we propose to use DNNs for estimating differences of phase between adjacent time-frequency bins. We show that convolutional neural networks are suitable for phase difference estimation, according to the theoretical relation between partial derivatives of STFT phase and magnitude. The estimated phase differences are used for reconstructing phase by solving a weighted least squares problem in a frame-by-frame manner. In contrast to existing DNN-based phase reconstruction methods, the proposed framework is causal and does not require any iterative procedure. The experiments showed that the proposed method outperforms existing online methods and a DNN-based method for phase reconstruction.

Deep-Learning-Assisted Sound Source Localization From a Flying Drone

Sound source localization from a flying drone is a challenging task due to the strong ego-noise from rotating motors and propellers as well as the movement of the drone and the sound sources. To address this challenge, we propose a deep-learning-based framework that integrates single-channel noise reduction and multichannel source localization. In this framework, we suppress the ego-noise and estimate a time–frequency soft ratio mask with a single-channel deep neural network (DNN). Then, we design two downstream multichannel source localization algorithms, based on steered response power (SRP-DNN) and time–frequency spatial filtering (TFS-DNN). The main novelty lies in the proposed TFS-DNN approach, which estimates the presence probability of the target sound at the individual time–frequency bins by combining the DNN-inferred soft ratio mask and the instantaneous direction of arrival (DOA) of the sound received by the microphone array. The time–frequency presence probability of the target sound is then used to design a set of spatial filters to construct a spatial likelihood map for source localization. By jointly exploiting spectral and spatial information, TFS-DNN robustly processes signals in short segments (e.g., 0.5 s) in dynamic and low signal-to-noise-ratio (SNR) scenarios (e.g., SNR −20 dB). Results on real and simulated data in a variety of scenarios (static sources, moving sources, and moving drones) indicate the advantage of TFS-DNN over competing methods, including SRP-DNN and the state-of-the-art TFS.

Modulation Recognition Using Signal Enhancement and Multistage Attention Mechanism

Robustness against noise is critical for modulation recognition (MR) approaches deployed in real-world communication systems. In MR systems, a corrupted signal is normally enhanced using low-level signal enhancement (SE) before signal classification (SC). Many existing approaches address signal distortion problems by compartmentalizing SE from SC. While those approaches allow for efficient development, they also dictate compartmentalized performance metrics, without feedback from the SC module. For example, SE modules are designed using perceptual signal quality metrics but not with SC in mind. To improve the effectiveness of SE on MR, this paper proposes a joint learning framework consisting of three cascaded modules: dual-channel spectrum fusion, SE, and SC. Instead of separately processing SE and SC, these three modules are integrated into one framework and jointly trained with a single recognition loss. In contrast to estimating clean signals, the SE module in the proposed joint learning framework is trained to predict a ratio mask and find important time-frequency bins for the SC module. We integrate a multistage attention mechanism into the framework to further increase the robustness. The multistage attention mechanism is deployed to strengthen the recognition-related features learned from context information in channel, time, and frequency domains. We evaluate the recognition performance of the proposed framework and its modules on two benchmark datasets: RadioML2016.10a and RadioML2016.10b. The experiment results show that the proposed joint learning framework outperforms the separate learning framework. Moreover, comparisons are performed with several existing learning-based MR methods in the literature. The proposed joint learning framework leads to significant performance improvement, especially for modulated signals corrupted by channel noise.

Simulating Spectral Kurtosis Mitigation against Realistic Radio Frequency Interference Signals

We investigate the effectiveness of the statistical radio frequency interference (RFI) mitigation technique spectral kurtosis () in the face of simulated realistic RFI signals. estimates the kurtosis of a collection of M power values in a single channel and provides a detection metric that is able to discern between human-made RFI and incoherent astronomical signals of interest. We test the ability of to flag signals with various representative modulation types, data rates, duty cycles, and carrier frequencies. We flag with various accumulation lengths M and implement multiscale , which combines information from adjacent time-frequency bins to mitigate weaknesses in single-scale . We find that signals with significant sidelobe emission from high data rates are harder to flag, as well as signals with a 50% effective duty cycle and weak signal-to-noise ratios. Multiscale with at least one extra channel can detect both the center channel and sideband interference, flagging greater than 90% as long as the bin channel width is wider in frequency than the RFI.

A Time-Frequency Bins Selection Pipeline for Direction-of-Arrival Estimation Using a Single Acoustic Vector Sensor

The case that multiple speakers are simultaneously active is usually regarded as adverse condition in the context of multi-source direction-of-arrival (DOA) estimation. Moreover, in adjacent sources scenarios, classifying DOA estimates by directions may merge adjacent sources and misclassify reverberation as sources, i.e., there is a mismatch between sources and reverberation. In order to address the open challenges mentioned above, this work introduces a pipeline for selecting reliable time-frequency (TF) bins and estimating multi-source DOAs using a single acoustic vector sensor (AVS). In the proposed pipeline, we show a viewpoint that the multiple simultaneously active sources condition can be exploited to select TF bins solely dominated by sources. The consistency of the distribution of intensity vectors (IVs) in multi-source scenarios is derived mathematically to support the proposed viewpoint. We address the mismatch between sources and reverberations in adjacent sources scenarios by focusing only on the direction with the highest intensity vector density (IVD), regardless of how many sources dominate the given TF window. Simulation and experimental results show that the proposed method can accurately and robustly estimate the multi-source DOAs in various scenarios.

SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Polyphonic Sound Event Localization and Detection

Sound event localization and detection (SELD) consists of two subtasks, which are sound event detection and direction-of-arrival estimation. While sound event detection mainly relies on time-frequency patterns to distinguish different sound classes, direction-of-arrival estimation uses amplitude and/or phase differences between microphones to estimate source directions. As a result, it is often difficult to jointly optimize these two subtasks. We propose a novel feature called Spatial cue-Augmented Log-SpectrogrAm (SALSA) with exact time-frequency mapping between the signal power and the source directional cues, which is crucial for resolving overlapping sound sources. The SALSA feature consists of multichannel log-spectrograms stacked along with the normalized principal eigenvector of the spatial covariance matrix at each corresponding time-frequency bin. Depending on the microphone array format, the principal eigenvector can be normalized differently to extract amplitude and/or phase differences between the microphones. As a result, SALSA features are applicable for different microphone array formats such as first-order ambisonics (FOA) and multichannel microphone array (MIC). Experimental results on the TAU-NIGENS Spatial Sound Events 2021 dataset with directional interferences showed that SALSA features outperformed other state-of-the-art features. Specifically, the use of SALSA features in the FOA format increased the F1 score and localization recall by 6% each, compared to the multichannel log-mel spectrograms with intensity vectors. For the MIC format, using SALSA features increased F1 score and localization recall by 16% and 7%, respectively, compared to using multichannel log-mel spectrograms with generalized cross-correlation spectra.

Multi-Level Time-Frequency Bins Selection for Direction of Arrival Estimation Using a Single Acoustic Vector Sensor

In the context of multi-source direction of arrival (DOA) estimation in an enclosed environment, the challenges include reverberation and overlapping of multiple simultaneous active sources. To address these interferences, the identification of time-frequency (TF) bins dominated by the sources signals is essential. In this work, we propose an intensity vector (IV) based TF bins selection technique for DOA estimation using a single acoustic vector sensor (AVS). The proposed technique involves multi-level inliers selection and outliers removal (MLISOR), which is implemented in three steps. In the first step, we derive the distribution of IVs and then select IVs using a norm metric. In the second step, the regions with the highest local IV density in each time frame are identified. In the third step, we cluster the IVs according to their directions and remove the outliers based on the member-to-centroid angle metric. Simulation results show that both the accuracy and the robustness of the proposed technique outperform the existing techniques. The indoor experimental results also verify that the proposed technique is effective and robust in practical situations.

A recursive expectation-maximization algorithm for speaker tracking and separation

The problem of blind and online speaker localization and separation using multiple microphones is addressed based on the recursive expectation-maximization (REM) procedure. A two-stage REM-based algorithm is proposed: (1) multi-speaker direction of arrival (DOA) estimation and (2) multi-speaker relative transfer function (RTF) estimation. The DOA estimation task uses only the time frequency (TF) bins dominated by a single speaker while the entire frequency range is not required to accomplish this task. In contrast, the RTF estimation task requires the entire frequency range in order to estimate the RTF for each frequency bin. Accordingly, a different statistical model is used for the two tasks. The first REM model is applied under the assumption that the speech signal is sparse in the TF domain, and utilizes a mixture of Gaussians (MoG) model to identify the TF bins associated with a single dominant speaker. The corresponding DOAs are estimated using these bins. The second REM model is applied under the assumption that the speakers are concurrently active in all TF bins and consequently applies a multichannel Wiener filter (MCWF) to separate the speakers. As a result of the assumption of the concurrent speakers, a more precise TF map of the speakers’ activity is obtained. The RTFs are estimated using the outputs of the MCWF-beamformer (BF), which are constructed using the DOAs obtained in the previous stage. Next, using the linearly constrained minimum variance (LCMV)-BF that utilizes the estimated RTFs, the speech signals are separated. The algorithm is evaluated using real-life scenarios of two speakers. Evaluation of the mean absolute error (MAE) of the estimated DOAs and the separation capabilities, demonstrates significant improvement w.r.t. a baseline DOA estimation and speaker separation algorithm.

Dynamically localizing multiple speakers based on the time-frequency domain

In this study, we present a deep neural network-based online multi-speaker localization algorithm based on a multi-microphone array. Following the W-disjoint orthogonality principle in the spectral domain, time-frequency (TF) bin is dominated by a single speaker and hence by a single direction of arrival (DOA). A fully convolutional network is trained with instantaneous spatial features to estimate the DOA for each TF bin. The high-resolution classification enables the network to accurately and simultaneously localize and track multiple speakers, both static and dynamic. Elaborated experimental study using simulated and real-life recordings in static and dynamic scenarios demonstrates that the proposed algorithm significantly outperforms both classic and recent deep-learning-based algorithms. Finally, as a byproduct, we further show that the proposed method is also capable of separating moving speakers by the application of the obtained TF masks.

Narrowband multi-source direction-of-arrival estimation in the spherical harmonic domain.

A conventional approach to wideband multi-source (MS) direction-of-arrival (DOA) estimation is to perform single source (SS) DOA estimation in time-frequency (TF) bins for which a SS assumption is valid. Such methods use the W-disjoint orthogonality (WDO) assumption due to the speech sparseness. As the number of sources increases, the chance of violating the WDO assumption increases. As shown in the challenging scenarios with multiple simultaneously active sources over a short period of time masking each other, it is possible for a strongly masked source (due to inconsistency of activity or quietness) to be rarely dominant in a TF bin. SS-based DOA estimators fail in the detection or accurate localization of masked sources in such scenarios. Two analytical approaches are proposed for narrowband DOA estimation based on the MS assumption in a bin in the spherical harmonic domain. In the first approach, eigenvalue decomposition is used to decompose a MS scenario into multiple SS scenarios, and a SS-based analytical DOA estimation is performed on each. The second approach analytically estimates two DOAs per bin assuming the presence of two active sources per bin. The evaluation validates the improvement to double accuracy and robustness to sensor noise compared to the baseline methods.

Exploiting the directional coherence function for multichannel source extraction

The desired speech detector plays an important role for controlling the speech distortion in spatial filtering based speech enhancement algorithms. However, the conventional complex coherence(CC) based algorithms can only distinguish the coherent speech and diffuse noise. To improve the performance on the scenarios that both the coherent interference and diffuse noise are present, we propose a directional coherence function(DCF) based detector in this paper. Based on a pair of complementary filters which can suppress the diffuse noise and the coherent interference respectively, the DCF is computed as the normalized correlation between the filters’ outputs. Meanwhile, the filters are solved by convex programming method and satisfy the constraints on speech distortionless and white noise gain(WNG). Consequently, the value of DCF will be close to 1 only for the desired speech dominated time-frequency(T-F) bins and much smaller than 1 for the noise or interference dominated T-F bins. To extract the desired speech, the DCF based Desired Speech Presence Probability(DSPP) is used to control the adaptation in general sidelobe canceler(GSC), and subsequently used as the post-filtering weight. Systematical experiments on several scenarios show that the proposed algorithm achieves significantly and consistently better noise suppression performance than the narrowband direction-of-arrival(DOA) estimates based algorithms.

The Effect of Partial Time-Frequency Masking of the Direct Sound on the Perception of Reverberant Speech

The perception of sound in real-life acoustic environments, such as enclosed rooms or open spaces with reflective objects, is affected by reverberation. Hence, reverberation is extensively studied in the context of auditory perception, with many studies highlighting the importance of the direct sound for perception. Based on this insight, speech processing methods often use time-frequency (TF) analysis to detect TF bins that are dominated by the direct sound, and then use the detected bins to reproduce or enhance the speech signals. The detection of bins dominated by the direct sound is typically based on an objective measure, such as the direct-to-reverberant ratio (DRR). However, the relation between the DRR in the TF bins and the spatial perception of the reverberant sound which is reproduced from these bins is still not clear. It is the aim of this paper to provide some insights into this relation, specifically for reverberant speech, focusing on bins with high DRR. This is performed using a listening experiment, where high DRR bins within a reverberant speech signal have been masked in the TF domain, based on various DRR thresholds. The results show that the percentage of high-DRR TF bins that were masked may better indicate the quality of spatial perception, compared to the specific value of the DRR threshold. The insights from this work could be incorporated into spatial audio techniques that reproduce the direct sound of reverberant speech, and potentially improve spatial perception. This was illustrated with an implementation of directional audio coding that was studied with an additional listening experiment supporting the previously described results.

Time-Frequency-Bin-Wise Linear Combination of Beamformers for Distortionless Signal Enhancement

In this paper, we address signal enhancement in underdetermined situations and propose new beamforming algorithms. Beamforming in (over)determined situations can successfully reduce noise signals without distortion of a desired signal, which is known to be a desirable property, especially for automatic speech recognition systems. Even in underdetermined situations, time-frequency (TF) masking attains outstanding performance in noise reduction, although it tends to generate artifacts. Integrating these two approaches to benefit from both their advantages, we here propose time-frequency-bin-wise switching (TFS) and time-frequency-bin-wise linear combination (TFLC) beamforming. In the proposed methods, we utilize the best combination of beamformers among multiple beamformers at each TF bin, each of which suppresses a particular combination of interferers. First, we propose a general formulation of signal enhancement employing multiple spatial filters. Then a joint optimization problem of designing the spatial filters and estimating the suitable weights to combine them is considered under a unified minimum variance criterion. Finally, we present efficient algorithms to solve the problem. In experiments, we used an objective criterion that quantifies the amount of signal distortion caused by the enhancement function and confirmed that the proposed methods effectively suppress interferers without distortion of the target signal.