Spatial Upsampling Research Articles

Time-Aligned Spatial Upsampling of Spherical Microphone Array Recordings

With the Spherical Array Interpolation by Time Alignment (SARITA) method, the authors introduced an approach for spatial upsampling of spherical microphone array (SMA) signals (T. Lübeck, J. M. Arend, and C. Pörschmann, IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1163–1174 [2023]). The basic idea of this method is to perform the interpolation after time-aligning adjacent microphone signals. The upsampled SMA signals can be represented as spherical harmonic coefficients of much higher spatial order than is possible with the sparsely measured signals. Instead of impulse responses, the method is now applied to SMA recordings. Binaural decoding of upsampled SMA recordings is compared technically and perceptually to a baseline Ambisonics decoding. The results show that the SARITA method can also be applied to low-order array recordings and significantly improves their binaural reproduction.

Continuous Space-Time Video Super-Resolution with Multi-Stage Motion Information Reorganization

Space-time video super-resolution (ST-VSR) aims to simultaneously expand a given source video to a higher frame rate and resolution. However, most existing schemes either consider fixed intermediate time and scale or fail to exploit long-range temporal information due to model design or inefficient motion estimation and compensation. To address these problems, we propose a continuous ST-VSR method to convert the given video to any frame rate and spatial resolution with Multi- s tage M otion information r eorganization (MsMr). To achieve time-arbitrary interpolation, we propose a forward warping guided frame synthesis module and an optical flow-guided context consistency loss to better approximate extreme motion and preserve similar structures among input and prediction frames. To realize continuous spatial upsampling, we design a memory-friendly cascading depth-to-space module. Meanwhile, with the sophisticated reorganization of optical flow, MsMr realizes more efficient motion estimation and motion compensation, making it possible to propagate information from long-range neighboring frames and achieve better reconstruction quality. Extensive experiments show that the proposed algorithm is flexible and performs better on various datasets than the state-of-the-art methods. The code will be available at https://github.com/hahazh/LD-STVSR .

Spatial Upsampling of Sparse Spherical Microphone Array Signals

We present a method for spatial upsampling of signals captured with spherical microphone arrays with a limited number of microphones. The upsampling is performed by adding virtual microphone signals using interpolation between the measured array signals. For interpolation, the differences in amplitude and time of neighbored signals are weighted separately. For this, the time differences are determined with a broadband cross-correlation. After the upsampling, the array signals can be represented as spherical harmonics (SH) coefficients at significantly higher orders. This work introduces the proposed upsampling procedure and presents a technical evaluation based on simulated and measured sound fields. Thereby, we focus on the binaural decoding of the upsampled SH signals. For perceptual evaluation, we present the results of a listening experiment evaluating the binaural reproduction of the upsampled signals compared to a baseline Ambisonics decoding and the state-of-the-art decoding with Magnitude Least-Squares optimization. Both the technical and perceptual evaluations consider only array impulse responses, but the algorithm could generally be applied to array streams. The evaluation shows that the proposed method significantly improves the binaural reproduction of low-order spherical microphone array captures compared to established methods.

Spatial up-sampling of HRTF sets using generative adversarial networks: A pilot study

Headphones-based spatial audio simulations rely on Head-related Transfer Functions (HRTFs) in order to reconstruct the sound field at the entrance of the listener’s ears. A HRTF is strongly dependent on the listener’s specific anatomical structures, and it has been shown that virtual sounds recreated with someone else’s HRTF result in worse localisation accuracy, as well as altering other subjective measures such as externalisation and realism. Acoustic measurements of the filtering effects generated by ears, head and torso has proven to be one of the most reliable ways to obtain a personalised HRTF. However this requires a dedicated and expensive setup, and is time-intensive. In order to simplify the measurement setup, thereby improving the scalability of the process, we are exploring strategies to reduce the number of acoustic measurements without degrading the spatial resolution of the HRTF. Traditionally, spatial up-sampling of HRTF sets is achieved through barycentric interpolation or by employing the spherical harmonics framework. However, such methods often perform poorly when the provided HRTF data is spatially very sparse. This work investigates the use of generative adversarial networks (GANs) to tackle the up-sampling problem, offering an initial insight about the suitability of this technique. Numerical evaluations based on spectral magnitude error and perceptual model outputs are presented on single spatial dimensions, therefore considering sources positioned only in one of the three main planes: Horizontal, median, and frontal. Results suggest that traditional HRTF interpolation methods perform better than the proposed GAN-based one when the distance between measurements is smaller than 90°, but for the sparsest conditions (i.e., one measurement every 120°–180°), the proposed approach outperforms the others.

Comparative evaluation of interpolation methods for the directivity of musical instruments

Measurements of the directivity of acoustic sound sources must be interpolated in almost all cases, either for spatial upsampling to higher resolution representations of the data, for spatial resampling to another sampling grid, or for use in simulations of sound propagation. The performance of different interpolation techniques applied to sparsely sampled directivity measurements depends on the sampling grid used but also on the radiation pattern of the sources themselves. Therefore, we evaluated three established approaches for interpolation from a low-resolution sampling grid using high-resolution measurements of a representative sample of musical instruments as a reference. The smallest global error on average occurs for thin plate pseudo-spline interpolation. For interpolation based on spherical harmonics (SH) decomposition, the SH order and the spatial sampling scheme applied have a strong and difficult to predict influence on the quality of the interpolation. The piece-wise linear, spherical triangular interpolation provides almost as good results as the first-order spline approach, albeit with on average 20 times higher computational effort. Therefore, for spatial interpolation of sparsely sampled directivity measurements of musical instruments, the thin plate pseudo-spline method applied to absolute-valued data is recommended and, if necessary, a subsequent modeling of the phase.

Bilateral Upsampling Network for Single Image Super-Resolution With Arbitrary Scaling Factors.

Single Image Super-Resolution (SISR) is essential for many computer vision tasks. In some real-world applications, such as object recognition and image classification, the captured image size can be arbitrary while the required image size is fixed, which necessitates SISR with arbitrary scaling factors. It is a challenging problem to take a single model to accomplish the SISR task under arbitrary scaling factors. To solve that problem, this paper proposes a bilateral upsampling network which consists of a bilateral upsampling filter and a depthwise feature upsampling convolutional layer. The bilateral upsampling filter is made up of two upsampling filters, including a spatial upsampling filter and a range upsampling filter. With the introduction of the range upsampling filter, the weights of the bilateral upsampling filter can be adaptively learned under different scaling factors and different pixel values. The output of the bilateral upsampling filter is then provided to the depthwise feature upsampling convolutional layer, which upsamples the low-resolution (LR) feature map to the high-resolution (HR) feature space depthwisely and well recovers the structural information of the HR feature map. The depthwise feature upsampling convolutional layer can not only efficiently reduce the computational cost of the weight prediction of the bilateral upsampling filter, but also accurately recover the textual details of the HR feature map. Experiments on benchmark datasets demonstrate that the proposed bilateral upsampling network can achieve better performance than some state-of-the-art SISR methods.

A Novel Macroblock-Level Filtering Upsampling Architecture for H.264/AVC Scalable Extension

The scalable extension of the H.264/AVC standard, also called H.264/AVC SVC standard or simply SVC standard uses spatial upsampling in the spatial scalability modes. This work presents a novel upsampling architecture designed for operation at macroblock level and dyadic upsampling ratio with QVGA as the base layer resolution and VGA as the enhancement resolution.The adoption of a macroblock-level solution translates into a more efficient use of hardware resources with savings of approximately 25% in the number of ALUTs and DLRs and using about two hundred times less memory bits, when compared to previously published works. The designed architecture was synthesized targeting four FPGAs: Altera Cyclone III and Stratix IV and Xilinx Spartan 3E and Virtex 4. The best throughput was reached by the Xilinx Virtex 4 device, with a processing rate of 506 VGA frames per second. The worst result was reached by the Xilinx Spartan 3E FPGA, with 249 VGA frames per second. All target FPGAs surpasses the necessary throughput to decode VGA videos in real time. This very high throughput is important especially when low power applications are considered, since with low operation frequencies (9.34MHz) it is possible to reach real time (30 frames per second) for all target FPGAs.

A Method for Spatial Upsampling of Voice Directivity by Directional Equalization

A Method for Spatial Upsampling of Voice Directivity by Directional Equalization

Correction to “Directional Equalization of Sparse Head-Related Transfer Function Sets for Spatial Upsampling” [Jun 19 1060-1071

In the paper “Directional Equalization of Sparse Head-Related Transfer Function Sets for Spatial Upsampling”, published in IEEE/ACM Transactions on Audio, Speech, andLanguage Processing (Volume: 27, Issue: 6, June 2019) [1],Eq. (2) and Eq. (6) are incorrect. In this Comment Correspondence we provide a correction of these Errors.

Directional Equalization of Sparse Head-Related Transfer Function Sets for Spatial Upsampling

Acquiring decent full-spherical sets of head-related transfer functions HRTFs based on a small number of measurements is highly desirable. For spatial upsampling, HRTF interpolation in the spatially continuous spherical harmonics SH domain is a common approach. However, the number of measured HRTFs limits the assessable SH order, resulting in order-limited HRTFs when transformed to the SH domain. Thus, the SH representation of sparse HRTF sets shows restricted spatial resolution and suffers from order-limitation errors. We present a method that reduces these errors by a directional equalization prior to the SH transform. This is done by a spectral division of each HRTF with a corresponding directional rigid sphere transfer function. The processing removes direction-dependent temporal and spectral components and, therefore, significantly reduces the spatial complexity of the HRTF set, allowing for an enhanced interpolation of HRTFs at reduced SH orders. Spatial upsampling is achieved by an inverse SH transform on an arbitrary dense sampling grid. A subsequent de-equalization by a spectral multiplication with the rigid sphere transfer function recovers the energy in higher spatial orders that was not inherent in the sparse HRTF set. For evaluation, HRTFs were calculated for various limited orders from sparse datasets and compared to a reference. The results show that the proposed method clearly outperforms common SH interpolation of HRTF spectra regarding the overall spectral and temporal structure as well as modeled localization performance.

A 3-D Sparse FIR Frustum Filter for Enhancing Broadband Plane Waves

A 3-D finite-extent impulse response (FIR) frustum filter having sparse coefficients is proposed for enhancing broadband plane waves (PWs) received by uniform aperture arrays. Motivated by the design technique of 1-D interpolated FIR filters, the proposed filter is realized by cascading a spatially-upsampled 3-D FIR prototype filter having a frustum-shaped passband and a 3-D FIR masking filter having a parallelepiped passband. Sparsity of the filter coefficients is achieved through the spatial upsampling of the coefficients of the prototype filter and by hard-thresholding the coefficients of both prototype and masking filters. The proposed 3-D FIR frustum filter saves 39.5% multiplications and 66.6% additions compared to the state-of-the-art 3-D FIR frustum filters without degrading the performance in enhancing broadband PWs.

Path‐space Motion Estimation and Decomposition for Robust Animation Filtering

AbstractRenderings of animation sequences with physics‐based Monte Carlo light transport simulations are exceedingly costly to generate frame‐by‐frame, yet much of this computation is highly redundant due to the strong coherence in space, time and among samples. A promising approach pursued in prior work entails subsampling the sequence in space, time, and number of samples, followed by image‐based spatio‐temporal upsampling and denoising. These methods can provide significant performance gains, though major issues remain: firstly, in a multiple scattering simulation, the final pixel color is the composite of many different light transport phenomena, and this conflicting information causes artifacts in image‐based methods. Secondly, motion vectors are needed to establish correspondence between the pixels in different frames, but it is unclear how to obtain them for most kinds of light paths (e.g. an object seen through a curved glass panel).To reduce these ambiguities, we propose a general decomposition framework, where the final pixel color is separated into components corresponding to disjoint subsets of the space of light paths. Each component is accompanied by motion vectors and other auxiliary features such as reflectance and surface normals. The motion vectors of specular paths are computed using a temporal extension of manifold exploration and the remaining components use a specialized variant of optical flow. Our experiments show that this decomposition leads to significant improvements in three image‐based applications: denoising, spatial upsampling, and temporal interpolation.