Loss Concealment Research Articles

Packet Loss Concealment Using Dual-Side Waveform Similarity Overlap-and-Add

Packet loss is one of the most essential problems in speech communication. It will cause the information loss and uncomfortable for listeners in voice over IP. This investment proposed an approach based on waveform similarity measure using overlap-and-Add algorithm. The waveform similarity overlap-and-add (WSOLA) technique is an effective algorithm to deal with packet loss concealment (PLC). For real-time time communication, the WSOLA algorithm is widely used to deal with the length adaptation and packet loss concealment of speech signal. Time scale modification of audio signal is one of the most essential research topics in data communication, especially in voice of IP (VoIP). Herein, we proposed the dual-side WSOLA that is derived by standard WSOLA. Instead of only exploitation one direction speech data, the proposed method will reconstruct the lost voice data according to the preceding and cascading voice. The dual-side WSOLA can use both the past and future speech signal waveform to reconstruction voice waveform of lost packet. The evaluations show that the quality of the reconstructed speech signal of the dual-side WSOLA is higher than that of the standard WSOLA and GWSOLA on different packet loss rate and length using the metrics: PESQ and MOS. The significant improvement is obtained by dual side information in the proposed method. The proposed dual-side waveform similarity overlap-and-add (DSWSOLA) outperforms the traditional approaches.

Multipath Adaptive Packet Dispersion for Voice Applications

Problem statement: The quality of Voice over IP applications is decided by the main factors like network capacity and technologies such as codec type, Packet Loss Concealment (PLC) and Forward Error Correction (FEC). Approach: In MPLS networks multiple Label Switched Paths (LSP) are established between MPLS ingress and egress nodes to enhance the network performance and QoS for subscribers. To enhance the Quality of Service, Multipath adaptive packet dispersion is proposed for voice applications in IP networks. Results: The trigger handler is proposed to check the balanced load in the system. If the network condition becomes unbalanced, the adaptive packet scheduler classifies the flows and routed to the best shortest paths with the help of proposed packet dispersion technique. Conclusion: The VoIP traffic is routed along the most adequate path that has enough resource to meet a given target QoS. Simulation results are reported to show the efficiency of the proposed technique for effective VOIP flows.

Hybrid low delay frame loss concealment in an MDCT based audio codec

By combining tonal-dominant and noise-dominant signal frame loss concealment (FLC) approaches, a hybrid low delay FLC method is proposed for an modified discrete cosine transform (MDCT) based codec. Based on the observations that the phase of the MDCT-MDST (modified discrete sine transform) coefficients of tonal-dominant signals decreases linearly with the increase of the frame index and the amplitude keeps unchanged, the tonal-dominant signal FLC approach uses the frame interpolation to estimate the phase and magnitude of the MDCT-MDST coefficients of the lost frame while the noise-dominant signal FLC method implements a modified shaped-noise insertion. Both objective and subjective test results show that the proposed technique provides better performance than the existing methods for music signals and voiced speech signals.

Robust Transmission of Audio Signals over the Internet: An Advanced Packet Loss Concealment for MP3-Based Audio Signals

This paper describes packet loss concealment methods for MP3 audio. The proposed methods are based on estimation of modified discrete cosine transform (MDCT) coefficients of the lost packets. The estimation of MDCT coefficients of lower dimensions is performed by switching two concealment methods: the sign correction method and the correlation-based method. The concealment methods are switched based on redundant side information calculated subband-by-subband for reducing MDCT prediction errors. Next, a method for improving estimation of MDCT coefficients of higher dimensions was proposed. The method estimates the absolute value and sign of an MDCT coefficient independently. The subjective evaluation experiment proved that both of the improvement methods for lower and higher dimensions effectively improved the subjective audio quality.

Frame loss concealment for stereoscopic video plus depth sequences

In this work, an efficient frame loss concealment algorithm for stereoscopic video plus depth sequences is proposed. To conceal an erroneous left color frame, we extrapolate motion vectors for the temporal error concealment. To conceal a left depth frame, we exploit the motion vectors of the corresponding left color frame. To conceal a right color frame, we employ a 3D image warping technique to determine matching pixels between inter-views, and perform the reconstruction based on the similarities of the motion vectors and the intensity differences of matching pixels. Finally, to conceal a right depth frame, we use the motion vectors of the right color frame and the left color frame. Simulation results show that the proposed algorithm is more effective than conventional algorithms in recovering erroneous color and depth frames. Moreover, it is also demonstrated that the proposed algorithm can be used for efficient virtual view synthesis in error prone environments.

Burst Packet Loss Concealment Using Multiple Codebooks and Comfort Noise for CELP-Type Speech Coders in Wireless Sensor Networks

In this paper, a packet loss concealment (PLC) algorithm for CELP-type speech coders is proposed in order to improve the quality of decoded speech under burst packet loss conditions in a wireless sensor network. Conventional receiver-based PLC algorithms in the G.729 speech codec are usually based on speech correlation to reconstruct the decoded speech of lost frames by using parameter information obtained from the previous correctly received frames. However, this approach has difficulty in reconstructing voice onset signals since the parameters such as pitch, linear predictive coding coefficient, and adaptive/fixed codebooks of the previous frames are mostly related to silence frames. Thus, in order to reconstruct speech signals in the voice onset intervals, we propose a multiple codebook-based approach that includes a traditional adaptive codebook and a new random codebook composed of comfort noise. The proposed PLC algorithm is designed as a PLC algorithm for G.729 and its performance is then compared with that of the PLC algorithm currently employed in G.729 via a perceptual evaluation of speech quality, a waveform comparison, and a preference test under different random and burst packet loss conditions. It is shown from the experiments that the proposed PLC algorithm provides significantly better speech quality than the PLC algorithm employed in G.729 under all the test conditions.

Intra-frame loss concealment based on directional extrapolation

A spatial loss concealment algorithm for reconstruction of the lost blocks in the I-frames of a video sequence is proposed. The algorithm is carried out in three steps. In the first step an edge directional analysis on the boundaries of the lost blocks is carried out. In the second step, the lost pixels are approximated from the available neighboring blocks of that boundary. In the third step, the derived approximations of all available boundaries are combined for final reconstruction of the lost pixels. Experimental results over an H.264/AVC codec show an improvement of about 0.5–1 dB over a similar spatial loss concealment algorithm. The subjective quality of loss concealed pictures is also evaluated through SSIM measure.

Network-aware perceptual error concealment method for H.264 video with side information

In order to improve the video quality of transmission with data loss, a spatial and temporal error concealment method was proposed, which considered both the state information of the network and the perceptual weight of the video content. The proposed method dynamically changed the reliability weight of the neighboring macroblock, which was used to conceal the lost macroblocks according to the packet loss rate of the current channel state. The perceptual weight map was utilized as side information to do weighted pixel interpolation and side-match based motion compensation for spatial and temporal error concealment, respectively. And the perceptual weight of the neighboring macroblocks was adaptively modified according to the perceptual weight of the lost macroblocks. Compared with the method used in H.264 joint model, experiment results show that the proposed method performs well both in subjective video quality and objective video quality, and increases the average peak signal-to-noise ratio (PSNR) of the whole frame by about 0.4 dB when the video bitstreams are transmitted with packets loss.

Streaming audio packet loss concealment based on sinusoidal frequency estimation in MDCT domain

A Packet Loss Concealment (PLC) method is proposed for audio streaming represented in the Modified Discrete Cosine Transform (MDCT) domain. MDCT is the dominant transform of perceptual audio coding and widely used in consumer electronics. The MDCT coefficients of a lost frame are reconstructed by our PLC method, which is based on audio sinusoidal model in MDCT domain. The sinusoid-like coefficients could be predicted by sinusoidal frequency estimation, and further optimization of computation is achieved by using sign property of sinusoids. In order to compare with other approaches, both subjective and objective evaluation tests are conducted. The experimental results show that the computational complexity and memory requirement of our approach are greatly reduced, while good quality is maintained.

The Design of VoIP Systems With High Perceptual Conversational Quality

This paper describes our work on real-time two-party and multi-party VoIP (voice-over-IP) systems that can achieve high perceptual conversational quality. It focuses on the fundamental understanding of conversational quality and its trade-offs among the design of speech codecs and strategies for network control, playout scheduling, and loss concealments. We have studied three key aspects that address the limitations of existing work and improve the perceptual quality of VoIP systems. Firstly, we have developed a statistical approach based on just-noticeable difference (JND) to significantly reduce the large number of subjective tests, as well as a classification method to automatically learn and generalize the results to unseen conditions. Using network and conversational conditions measured at run time, the classifier learned helps adjust the control algorithms in achieving high perceptual conversational quality. Secondly, we have designed a cross-layer speech codec to interface with the loss-concealment and playout scheduling algorithms in the packet-stream layer in order to be more robust and effective against packet losses. Thirdly, we have developed a distributed algorithm for equalizing mutual silences and an overlay network for multi-party VoIP systems. The approach leads to multi-party conversations with high listening only speech quality and balanced mutual silences.

Guest Editors’ Introduction: Advances in Interactive Digital Entertainment Technologies

With the significant development of digital technologies in recent years, we are seeing an increasing number of applications of these technologies, in particular in the entertainment domain. They may include computer games, e-learning, high- definition and interactive TVs, and virtual environments. The development of these applications typically involves the integration of existing technologies as well as the development of new technologies. The first International Workshop on Digital Entertainment Technologies 2008 (IDET’08) was held at Lanzhou University, China, in July 2008, in conjunction with the first IEEE International Conference on Ubi-media Computing (U-Media’08). This workshop was an initial effort to review various technological issues and challenges in digital entertainments. A special emphasis was on issues that are relevant to or supporting the dynamic interactions between users and applications. This special issue collects the extended version of some of the best papers presented in IDET’08 and relevant keynote papers presented in U-Media’08. This special issue includes six papers covering some of the recent technological advances in digital entertainments. These papers can be roughly divided into three groups. The first group of two papers addresses networking issues of digital entertainments. The article by Wah and Sat (of University of Illinois, Urbana-Champaign, U.S.A.) describes the authors’ work on real-time VoIP (voice over-IP) systems that can achieve high perceptual conversational quality. It focuses on the fundamental understanding of conversational quality and its trade-offs among the design of speech codecs and strategies for network control, playout scheduling, and loss concealments. The article by Ye, Li, and Chen (of City University of Hong Kong, Hong Kong) presents an adaptive algorithm called “SPF-A*” for searching multimedia files in heterogeneous mobile P2P network environments in order for these multimedia resources to be transmitted more effectively. The second group of two papers addresses content analysis issues of digital entertainments. The article by Xu, Cheng, Zhang, Zhang, and Lu (of Chinese Academy of Sciences, China) presents a generic multi-layer and multi-modal framework for sports video analysis. It first introduces mid-level audio/visual features that are able to bridge the semantic gap between low-level features and high-level understanding and then discusses emerging applications on editorial content creation and content enhancement/adaptation in sports video analysis, including event detection, sports MTV generation, automatic broadcast video generation, tactic analysis, player action recognition, virtual content insertion, and mobile sports video adaptation. The article by Don and Ivrissimtzis (of University of Durham, United Kingdom) describes the use of multiple pens to resolve ambiguity and improve usability in a sketch recognition environment. User experiments in representing drawing modes using separate colored pens demonstrate greater user understanding and less recognition errors than in a system using textual descriptions with a single pen. The last group of two papers addresses issues that are related to computer gaming in digital entertainments. The article by Cao, Glukhova, Klamma, and Renzel (of RWTH Aachen University, Germany) presents a community success model (CSM) based on the DeLone & McLean IS Success Model to measure community satisfaction. CSM combines quantitative data collected through automatic monitoring of service usage with qualitative data extracted from automatically generated questionnaires, and CSM is applied to measure the results of a regular lab course as a proof of concept. The article by Li, Zhao, Lam, and Lau (of University of Durham, United Kingdom, and City University of Hong Kong, Hong Kong) presents an interactive method for simulating flexible connectors by combining dynamic constraints and impulse-based simulation. This method can be used to support interactive gaming or animation. We would like to thank a lot of people who worked behind the scene to make this special issue possible. First, we would like to thank George Sun, the Chief Executive Editor of the Journal of Multimedia, for his kind support of this special issue. Second, we would like to thank all the reviewers, including the workshop program committee members and those who helped select the papers for this special issue, for their professional reviews. Finally, we would like to thank all the authors who contributed their work to IDET 2008 and to this special issue.

Supplementary loss concealment technique for image transmission through data hiding

This paper addresses a loss concealment technique of image transmission in error prone environments. Imperfect transmission of imagery results in the loss of lines or blocks. Previous concealment techniques utilize the spatial, spectral, or temporal redundancy of the imagery. We propose a supplementary loss concealment technique that uses a data-hiding method. We divide an image into 8?×?8 pixels blocks and generate block description information of each block. The block description information is inserted into other blocks in the same image through LSB-based data hiding. Missing lines or blocks during transmission are restored by extracting this block description information. The technique to resist against lossy compression is also considered. We have performed a simulation to show the outstanding performance of the proposed technique in comparison with other loss concealment methods.

Performance Study of Objective Speech Quality Measurement for Modern Wireless-VoIP Communications

Wireless-VoIP communications introduce perceptual degradations that are not present with traditional VoIP communications. This paper investigates the effects of such degradations on the performance of three state-of-the-art standard objective quality measurement algorithms--PESQ, P.563, and an extended E-model. The comparative study suggests that measurement performance is significantly affected by acoustic background noise type and level as well as speech codec and packet loss concealment strategy. On our data, PESQ attains superior overall performance and P.563 and E-model attain comparable performance figures.

An Efficient Shared Adaptive Packet Loss Concealment Scheme through 1-Port Gateway System for Internet Telephony Service

In this letter, we propose a shared adaptive packet loss concealment scheme for the high quality guaranteed Internet telephony service which connects multiple users. In order to recover packet loss efficiently in the all-IP based convergence environment, we provide a robust signal recovery scheme which is based on the shared adaptive both-side information utilization. This scheme is provided according to the average magnitude variation across the frames and the pitch period replication on the 1-port gateway (G/W) system. The simulated performance demonstrates that the proposed scheme has the advantages of low processing times and high recovery rates in the all-IP based ubiquitous environment.

Interpolation of Lost Speech Segments Using LP-HNM Model With Codebook Post-Processing

This paper presents a method for interpolation of lost speech segments. The interpolation method can be used for packet loss concealment in voice communication over mobile phones, for voice over IP or for restoration of lost segments in speech recordings. The interpolation method employs a combination of a linear prediction (LP) model of the spectral envelope and a harmonic noise model (HNM) of the excitation of speech. The speech interpolation problem is transformed to the modeling and interpolation of the trajectories of LP parameters and the amplitude, phase and harmonicity of HNM tracks of speech excitation. In particular, the interpolation of harmonicity results in a smooth transition from voiced to unvoiced speech and vice versa. Crucially, the proposed interpolation method does not suffer from the consequences of zero-excitation of conventional autoregressive (AR) interpolation. Different combinations of linear and autoregressive interpolation methods are evaluated for the estimation of the time-varying parameters of LP-HNM tracks. Furthermore, a post-processing codebook mapping, employed to enhance the interpolation of the spectral envelope of speech, results in improved output quality for longer length speech gaps. For different packet loss rates and patterns of distributions of missing speech gaps, the proposed interpolation methods are evaluated and compared with popular AR-based interpolation methods and the speech packet recovery method specified in the ITU G.711 standard, as a reference. The evaluation results show that the proposed methods substantially improve the restoration of formants and harmonic tracks and consistently results in significant performance gain and improved perceptual quality of speech.

Audio Packet Loss Concealment in a Combined MDCT-MDST Domain

Audio streaming applications have become very popular in recent years, owing to their low cost and convenience. However, during network congestions, data packets are often delayed or discarded, creating an annoying gap in the streamed media. This letter presents a new approach to audio packet loss concealment designed for MPEG-audio streaming applications. In a previous work, we introduced a receiver-based concealment algorithm based on applying the gapped-data amplitude and phase estimation (GAPES) interpolation algorithm in the discrete short-time Fourier transform (DSTFT) complex domain and obtained better results compared to past methods. The current approach applies the same algorithm on a different complex domain, formed from combining the modified discrete cosine transform (MDCT) domain as its real part and the modified discrete sine transform (MDST) domain as the imaginary part. The new approach significantly reduces the complexity demands while maintaining similar high-quality results.

Packet Loss Concealment for G.722 using Side Information with Application to Voice over Wireless LANs

The G.722 wideband speech codec offers higher quality and better naturalness than G.711, is low in com- plexity, has low delay, and tandems well with other codecs. This makes it an attractive codec for Voice over IP and Voice over Wireless LANs. However, loss of a G.722 coded speech frame results in a mismatch of the encoder/decoder states that affects the decoding of subsequent correctly received frames. We present a new packet loss concealment (PLC) method for G.722 and propose an efficient approach to sending the side information to improve concealment of the lost frame and to resynchronize the encoder and decoder. The proposed method greatly improves the robustness of the G.722 codec to packet losses for Voice over Wireless LANs. Wideband Perceptual Evaluation of Speech Quality Mean Opinion Score (WB-PESQ MOS) is used as the quality metric for evaluating the performance of the PLC schemes.

Payload length and rate adaptation for multimedia communications in wireless LANs

We provide a theoretical framework for cross-layer design in multimedia communications to optimize single-user throughput by selecting the transmitted bit rate and payload size as a function of channel conditions for both additive white Gaussian noise (AWGN) and Nakagami-m fading channels. Numerical results reveal that careful payload length adaptation significantly improves the throughput performance at low signal to noise ratios (SNRs), while at higher SNRs, rate adaptation with higher payload lengths provides better throughput performance. Since we are interested in multimedia applications, we do not allow retransmissions in order to minimize latency and to reduce congestion on the wireless link and we assume that packet loss concealment will be used to compensate for lost packets. We also investigate the throughput and packet error rate performance over multipath frequency selective fading channels for typical payload sizes used in voice and video applications. We explore the difference in link adaptation thresholds for these payload sizes using the Nafteli Chayat multipath fading channel model, and we present a link adaptation scheme to maximize the throughput subject to a packet error rate constraint.

Short- and Long-Term Packet Loss Behavior: Towards Speech Quality Prediction for Arbitrary Loss Distributions

A speech-quality-oriented classification of packet loss distributions is proposed according to both the short- and long-term loss behavior. While the short-term behavior (microscopic loss behavior) relates to the effect of packet loss on the coder and packet loss concealment performance, the long-term loss behavior (macroscopic loss behavior) is defined so that it reflects the loss behavior that ultimately leads to speech quality that perceptively changes over time. Based on this classification, different parametric (objective) modeling approaches for predicting speech quality are discussed. To this aim, a packet loss averaging approach is presented for modeling speech quality under short-term loss. Starting from this model, two different ways for predicting speech quality under long-term-dependent packet loss are analyzed and compared to auditory (subjective) test results: quality prediction based on the averaging at packet trace level as provided, for example, by the E-model (2005), and the prediction based on the time-averaging of estimated instantaneous quality profiles, as suggested, for example, by L. Gros and N. Chateau (2001) (1998). From this comparison, the suitability of the different approaches for network planning are discussed, and their limitations in case of particular loss distributions are pointed out

Hidden Markov model-based packet loss concealment for voice over IP

As voice over IP proliferates, packet loss concealment (PLC) at the receiver has emerged as an important factor in determining voice quality of service. Through the use of heuristic variations of signal and parameter repetition and overlap-add interpolation to handle packet loss, conventional PLC systems largely ignore the dynamics of the statistical evolution of the speech signal, possibly leading to perceptually annoying artifacts. To address this problem, we propose the use of hidden Markov models for PLC. With a hidden Markov model (HMM) tracking the evolution of speech signal parameters, we demonstrate how PLC is performed within a statistical signal processing framework. Moreover, we show how the HMM is used to index a specially designed PLC module for the particular signal context, leading to signal-contingent PLC. Simulation examples, objective tests, and subjective listening tests are provided showing the ability of an HMM-based PLC built with a sinusoidal analysis/synthesis model to provide better loss concealment than a conventional PLC based on the same sinusoidal model for all types of speech signals, including onsets and signal transitions