Head-related Transfer Functions Personalization Research Articles

Grid-agnostic personalized head-related transfer function modeling with neural fields

The spatial filtering effect brought on by sound propagation from the sound source to the outer ear is referred to as the head-related transfer function (HRTF). The personalization of HRTF is essential to enhance the personalized immersive audio experience in virtual and augmented reality. Our work aims to employ deep learning to predict the customized HRTF from anthropometric measurements. However, existing measured HRTF databases each employ a different geographic sampling, making it difficult to combine these databases into training data-hungry deep learning methods while each of them only contains dozens of subjects. Following our previous work, we use a neural field, a neural network that maps the spherical coordinates to the magnitude spectrum to represent each subject’s set of HRTFs. We constructed a generative model to learn the latent space across subjects using such a consistent representation of HRTF across datasets. In this work, by learning the mapping of the anthropometric measurements to the latent space and then reconstructing the HRTF, we further investigate the neural field representation to carry out HRTF personalization. Thanks to the grid-agnostic nature of our method, we are able to train on combined datasets and even validate the performance on grids unseen during training.

Universal global compact representation of head-related transfer functions with hemi-spherical harmonics and conformal mapping

The head-related transfer function(HRTF) is an essential part of spatial auditory display systems. In recent years, numerous HRTF databases are established to include human subjects' measurements, which enabled data-driven research projects such as HRTF prediction and personalization. However, in most cases, each HRTF database has its own unique measuring standard and source direction grid set. It's difficult to merge and efficiently represent the global HRTF information across multiple HRTF databases. Current presentation methods, such as principle component analysis (PCA) and spherical harmonics transform (SHT), are constrained by the source grid layout and regularization errors, which lays burdens for common feature extraction from different sets of HRTF measurements. In this work, we propose a novel approach for the global compact representation of HRTFs across different measuring standards, using hemispherical harmonics (HSH) and conformal mapping. The method takes into account the spatial domain covered by the typical HRTF measurements and proves to be less contained by the source direction grids. Both numerical and auditory model experiments are performed, to examine the representation error and consistency when including multiple HRTF databases with different measuring standards and grid layouts. In continuing work, we are using this method as a pre-processing approach for HRTF personalization utilizing multiple HRTF databases.

Head-Related Transfer Function Personalization Based on Modified Sparse Representation with Matching in a Database of Chinese Pilots

A new method of head-related transfer function (HRTF) personalization was built to optimize new user matching in the HRTF database to enhance 3D virtual hearing. The method of modified sparse representation (MSR) treated the HRTF personalization for an incomer as the task of finding similar subjects in the HRTF database with sparse representation (SR) of the incomer’s anthropometric features under the assumption that the equivalent sparse combination expresses the same anthropometric features and HRTFs for virtual hearing. The correlations among principal component coefficients and all anthropometric parameters were calculated after principal component analysis of the HRTF database for the same orientation. A derived database of anthropometric parameters was reconstructed, and SR was performed for all subjects in the original HRTF database. When the SR of the incomer was best matched with that of a subject in the derived database, the HRTF of the matched subject was used for the incomer’s 3D virtual hearing. The effect of MSR HRTF personalization was evaluated by spectral distortion. The results showed that the proposed method performed better than the others with the Chinese pilots database and the CIPIC database, especially in the frequency range of 0–8 kHz.

Personalization of Head-Related Transfer Function Based on Sparse Principle Component Analysis and Sparse Representation of 3D Anthropometric Parameters

Personalization of head-related transfer functions (HRTFs) is an essential task in the research on virtual hearing. In this paper, a method based on sparse principal component analysis (SPCA) and sparse representation (SR) was proposed to personalize HRTFs. The fundamental assumption is that an equivalent sparse combination can express the same anthropometric parameters. SPCA was first used to reduce the dimensionality of three-dimensional anthropometric parameters, and the reduced physiological parameters were used to reconstruct a physiological parameter database. SR was performed on the reconstructed physiological parameters of all subjects. For each subject, SR was used on the anthropometric parameters that were consistent with those in the reconstruction database. The matching pursuit algorithm was used to obtain the subjects in the database that had the same SR as the subject, and the HRTFs of the matched subject were used as the HRTFs of the new subject. The effect of the proposed method was evaluated by spectral distortion. The results showed that the method did better than others regardless of whether the Chinese pilots’ database or the CIPIC database was used, especially in the 0–8 kHz bandwidth.

Reinforcement-Learning-Based Personalization of Head-Related Transfer Functions

Reinforcement-Learning-Based Personalization of Head-Related Transfer Functions

Personalization of head-related transfer functions in the median plane based on spectral correction with pinna angle

Head-related transfer functions (HRTFs) are studied to realize high-precision sound field reproduction systems. These systems usually utilize the HRTFs of other listeners because self-measurement of one’s own HRTFs is a heavy burden. The localized sound accuracy of the listener, especially in the median plane, is degraded when the systems utilize the HRTFs of other listeners, which have different characteristics from the listener’s own ones. Thus, simple personalization of HRTFs is necessary for solving this problem. In this research, we focus on personalization of HRTFs in the median plane on the basis of spectral correction with pinna angles as anthropometric parameters. We first made analyses of the relationship between pinna angles and the spectral envelopes of HRTFs in the median plane through an experiment with a dummy head with a replicated pinna made from silicone. From these analyses, the spectral envelopes were confirmed to be affected by pinna angles. We then proposed a method for personalization of HRTFs on the basis of spectral correction with the pinna angles. Objective and subjective evaluation experiments were carried out in order to demonstrate the effectiveness of the proposed method. As a result, we confirmed improvement in the performance of sound image localization.

Head-related transfer function personalization based on spectral notch estimation with stereo images of pinna

Three-dimensional (3-D) sound field reproduction systems such as surround sound reproduction systems are rapidly being developed in recent few years. However, these systems need a lot of space for the placement of loudspeakers. For this reason, a binaural system has been studied as one of the systems, which controls the sound image localization with the head-related transfer functions (HRTFs) and just by using only a headphones. However, the individual differences in body features would cause the variations of HRTFs which greatly affect the direction perception. For this reason, a method based on the spectral notch estimation from the pinna shape has been studied for the personalization of HRTFs. In this study, we propose a new method, in which the depth information used for the spectral notch estimation is measured with stereo images of the pinna. Two cameras are utilized together as a stereo camera to take the disparate stereo images so that we can use the disparity of them to measure the depth information of the pinna. As a result of an evaluation experiment, we could reduce the cost of actual measurement and the burden to the subjects by the proposed method.

Auditory navigation with a tubular acoustic model for interactive distance cues and personalized head-related transfer functions

This paper presents a novel spatial auditory display that combines a virtual environment based on a Digital Waveguide Mesh (DWM) model of a small tubular shape with a binaural rendering system with personalized head-related transfer functions (HRTFs) allowing interactive selection of absolute 3D spatial cues of direction as well as egocentric distance. The tube metaphor in particular minimizes loudness changes with distance, providing mainly direct-to-reverberant and spectral cues. The proposed display was assessed through a target-reaching task where participants explore a 2D virtual map with a pen tablet and hit a sound source (the target) using auditory information only; subjective time to hit and traveled distance were analyzed for three experiments. The first one aimed at assessing the proposed HRTF selection method for personalization and dimensionality of the reaching task, with particular attention to elevation perception; we showed that most subjects performed better when they had to reach a vertically unbounded (2D) rather then an elevated (3D) target. The second experiment analyzed interaction between the tube metaphor and HRTF showing a dominant effect of DWM model over binaural rendering. In the last experiment, participants using absolute distance cues from the tube model performed comparably well to when they could rely on more robust, although relative, intensity cues. These results suggest that participants made proficient use of both binaural and reverberation cues during the task, displayed as part of a coherent 3D sound model, in spite of the known complexity of use of both such cues. HRTF personalization was beneficial for participants who were able to perceive vertical dimension of a virtual sound. Further work is needed to add full physical consistency to the proposed auditory display.

Personalization of head-related transfer functions (HRTF) based on automatic photo-anthropometry and inference from a database

A method is presented to personalize HRTFs based on anthropometric features obtained from digital portraits of a subject through computer vision, using the active shape models (ASM) algorithm. The method presented here is automatic, advancing previous work, and uses a simple more efficient 2D image model to represent the subject. The ASM is trained beforehand on a data set of manually annotated images of other subjects, and then used to automatically recognize anthropometric features of the ears, head, and torso, on photographs of the given subject. Anthropometric parameters in metric linear units are obtained from the image model as a function of camera settings, field of view, and distance to the subject. Anthropometric data are finally used to personalize HRTFs for the subject, by inference from the CIPIC HRTF database, containing anthropometric and HRTF data for a number of subjects, using the method of best anthropometric match selection. Results are analyzed, concluding that personalization of HRTFs can be achieved by automatic photo-anthropometry through computer vision algorithms, and inference of HRTFs from a database. Results were evaluated using two definitions of acoustic spectral distortion, as comparative measures of performance: one in magnitude only, previously proposed, and a novel one in magnitude and phase. However, evidence is also presented suggesting that these performance metrics need to be improved further, and that current anthropometric and HRTF databases might still be insufficient to provide more satisfactory results, pointing out to the need for more research in this area.

Usage of Spectral Distortion for Objective Evaluation of Personalized HRTF in the Median Plane

Measuring the head-related transfer functions (HRTFs) for each subject is a complex process. Therefore, it is necessary to develop procedures that allow the estimation of personalized HRTFs. It is common to estimate the weights of the principal component analysis (PCA) of a group of subjects based on some anthropometric parameters using multivariable regression modelling. Moreover, to objectively evaluate the goodness of fit between the original HRTFs and the personalized ones, the spectral distortion (SD) is usually used too. However, its suitability in the median plane, in which the spectral profiles are crucial to localize a sound source, has not yet been demonstrated. This paper analyses the validity of the SD as a measure of the quality of the HRTF personalization in the median plane, from the localization point of view. The HRTFs were modelled from the weights estimated by multiple linear regression and artificial neural networks (ANNs). The SD was used to compare the HRTFs measured with those estimated. Likewise, the level of fitting accuracy of characteristic resonance and notches in the median plane was also compared. Despite the fact that the SD scores of ANNs are lower than those of the multiple linear regression and are similar to those reported by other studies, the errors obtained from analysing both central frequencies and levels for resonance and notches could be discriminated.

Personalization of head-related transfer functions in the median plane based on the anthropometry of the listener's pinnae.

A listener's own head-related transfer functions (HRTFs) are required for accurate three-dimensional sound image control. The HRTFs of other listeners often cause front-back confusion and errors in the perception of vertical angles. However, measuring the HRTFs of all listeners for all directions of a sound source is impractical because the measurement requires a special apparatus and a lot of time. The present study proposes a method for estimating the appropriate HRTFs for an individual listener. The proposed method estimates the frequencies of the two lowest spectral notches (N1 and N2), which play an important role in vertical localization, in the HRTF of an individual listener by anthropometry of the listener's pinnae. The best-matching HRTFs, of which N1 and N2 are the closest to the estimates, are then selected from an HRTF database. In order to examine the validity of the proposed method, localization tests in the upper median plane were performed using four subjects. The results revealed that the best-matching HRTFs provided approximately the same performance as the listener's own HRTFs for the target directions of the front and rear for all four subjects. For the upper target directions, however, the performance of the localization for some of the subjects decreased.

Modeling individual HRTF tensor using high-order partial least squares

A tensor is used to describe head-related transfer functions (HRTFs) depending on frequencies, sound directions, and anthropometric parameters. It keeps the multi-dimensional structure of measured HRTFs. To construct a multi-linear HRTF personalization model, an individual core tensor is extracted from the original HRTFs using high-order singular value decomposition (HOSVD). The individual core tensor in lower-dimensional space acts as the output of the multi-linear model. Some key anthropometric parameters as the inputs of the model are selected by Laplacian scores and correlation analyses between all the measured parameters and the individual core tensor. Then, the multi-linear regression model is constructed by high-order partial least squares (HOPLS), aiming to seek a joint subspace approximation for both the selected parameters and the individual core tensor. The numbers of latent variables and loadings are used to control the complexity of the model and prevent overfitting feasibly. Compared with the partial least squares regression (PLSR) method, objective simulations demonstrate the better performance for predicting individual HRTFs especially for the sound directions ipsilateral to the concerned ear. The subjective listening tests show that the predicted individual HRTFs are approximate to the measured HRTFs for the sound localization.

System for automatic personalization of head-related transfer functions based on computer vision, photo-anthropometry, and inference from a database

A software system and method for automatic personalization of head related transfer functions (HRTF) is presented. The system pursues the objective of personalizing HRTF using the anthropometry of the subject, as proposed originally by [Zotkin et al. WASPAA 2003, 157–160], which in this work, is measured automatically from a set of photographs of the subject. The system operates in three stages. First, a computer vision algorithm, known as active shape models, is used over the portraits to recognize and adjust geometric form profiles of the ears, head, and torso of the subject. Then, anthropometry is performed by measurement of pixel distances between specific points in the model, and then converted into metric units. Finally, HRTF are estimated from the anthropometry using a choice of three methods, whose performance is compared: (1) HRTF selection of the best anthropometric match in the CIPIC database, (2) HRTF synthesis by multiple linear regression and principal component analysis, and (3) HRTF synthesis using an artificial neural network. Results are analyzed, concluding that automatic personalization of HRTF is attainable automatically from the subject portraits, using computer vision and inference from a database. Further analysis, however, reveals the need for more complete HRTF and anthropometric databases.

Toward optimal functional representation of head-related transfer functions in the horizontal plane

Head-related transfer function (HRTF) individualization using principle component analysis (PCA) modeling rely on the empirical data to reduce HRTF dimensionality for an optimal representation and to achieve HRTF personalization by tuning the model weights with the subject anthropometric parameters. However, for these representations, the basis is discrete and data dependent, which can limit its usefulness in universal HRTF representation. This paper studies the optimal functional representation of magnitude HRTF of 45 subjects for sound sources in the horizontal plane. We first use circular harmonics to extract the subject-independent HRTF angular dependence. The remaining spectral components of 45 subjects are then modeled by PCA and two standard functions, i.e., Fourier series and Fourier Bessel series. The metric to evaluate the model efficiency is the expansion weights cumulative variance. We identify that individual magnitude HRTFs over 20 kHz range could be modeled adequately well by a linear combination of only 19 Fourier series; this is a near optimal representation in comparison with the statistical PCA model. Further analysis of the model weights with subjective anthropometric measurements will provide a promising method for HRTF individualization, especially considering the nature of data independent continuous basis functions employed in the proposed functional representation.

The role of spatial detail in sound-source localization: Impact on head-related transfer function modeling and personalization

Current techniques designed to personalize generic head-related transfer functions (HRTFs) have some capacity to quickly customize spatial auditory displays, but these techniques generally fall short of the level of realism and performance provided by fully individualized HRTF measurements. This residual performance deficit reflects inaccuracies due to vast amounts of spatial and spectral variation that occurs across the measured HRTFs of individual listeners. Some of this variation encodes perceptually-important directional information, but a substantial proportion does not. Kulkarni and Colburn (1998) showed that perceptually irrelevant spectral variation could be eliminated by smoothing the HRTF magnitude with a truncated Fourier-series expansion. The present study investigates a related method for smoothing the spatial variation contained in the HRTF by utilizing a truncated spherical harmonic expansion. The impact of spatial smoothing was evaluated by comparing localization performance with individualized HRTFs which were fully represented or had various degrees of spatial smoothing. Results indicate that a highly-smoothed fourth-order spherical harmonic representation can produce localization accuracy comparable to that of a full individualized HRTF. Analysis of the resulting simplified HRTF representations also uncovered a number of interesting relationships across different individuals which may provide new insights for the development of future HRTF personalization and estimation techniques.

HRTF personalization based on artificial neural network in individual virtual auditory space

The synthesis of individual virtual auditory space (VAS) is an important and challenging task in virtual reality. One of the key factors for individual VAS is to obtain a set of individual head related transfer functions (HRTFs). A customization method based on back-propagation (BP) artificial neural network (ANN) is proposed to obtain an individual HRTF without complex measurement. The inputs of the neural network are the anthropometric parameters chosen by correlation analysis and the outputs are the characteristic parameters of HRTFs together with the interaural time difference (ITD). Objective simulation experiments and subjective sound localization experiments are implemented to evaluate the performance of the proposed method. Experiments show that the estimated non-individual HRTF has small mean square error, and has similar perception effect to the corresponding one obtained from the database. Furthermore, the localization accuracy of personalized HRTF is increased compared to the non-individual HRTF.