Video Degradation Research Articles

Decoupling Degradations with Recurrent Network for Video Restoration in Under-Display Camera

Under-display camera (UDC) systems are the foundation of full-screen display devices in which the lens mounts under the display. The pixel array of light-emitting diodes used for display diffracts and attenuates incident light, causing various degradations as the light intensity changes. Unlike general video restoration which recovers video by treating different degradation factors equally, video restoration for UDC systems is more challenging that concerns removing diverse degradation over time while preserving temporal consistency. In this paper, we introduce a novel video restoration network, called D2RNet, specifically designed for UDC systems. It employs a set of Decoupling Attention Modules (DAM) that effectively separate the various video degradation factors. More specifically, a soft mask generation function is proposed to formulate each frame into flare and haze based on the diffraction arising from incident light of different intensities, followed by the proposed flare and haze removal components that leverage long- and short-term feature learning to handle the respective degradations. Such a design offers an targeted and effective solution to eliminating various types of degradation in UDC systems. We further extend our design into multi-scale to overcome the scale-changing of degradation that often occur in long-range videos. To demonstrate the superiority of D2RNet, we propose a large-scale UDC video benchmark by gathering HDR videos and generating realistically degraded videos using the point spread function measured by a commercial UDC system. Extensive quantitative and qualitative evaluations demonstrate the superiority of D2RNet compared to other state-of-the-art video restoration and UDC image restoration methods.

Inverse-tone-mapped HDR video quality assessment: A new dataset and benchmark

The accurate quality assessment for inverse-tone-mapped videos (ITMVs) has become increasingly important due to the ongoing rapid development of inverse tone mapping (ITM) algorithms. We observed that most existing video quality assessment (VQA) datasets and objective VQA methods are based on two underlying assumptions: the videos are in standard dynamic range (SDR) domain and the distortions are introduced in the video degradation process. However, ITMVs are videos in high dynamic range (HDR) domain, and are accompanied by distortions introduced during the enhancement process, which are distinct from those in degradation process. We then pose the question: Can existing VQA methods objectively assess the quality of ITMVs? To address this question and foster the advancement of VQA methods, we contribute the first publicly available dataset specifically tailored for assessing the quality of ITMVs. The dataset is called “VQA4ITM”, which is an acronym for “Video Quality Assessment for Inverse Tone Mapping”. Specifically, 30 distortion-free, professional 4K Rec. 2020 10 bit 50fps HDR source videos and 12 ITM methods were carefully selected, and a total of 360 ITMVs were generated in the end. A subjective experiment was conducted in a professional HDR viewing environment and 6,098 valid quality scores were finally gathered to calculate the mean opinion score (MOS) of each ITMV. Furthermore, we perform benchmark experiments on the VQA4ITM dataset, revealing that existing VQA methods are unable to fairly assess the quality of ITMVs. The inadequate performance of existing VQA methods on the VQA4ITM dataset highlights the challenging nature of our dataset, as well as the imperative need for the development of more effective VQA algorithms in the context of ITM. To facilitate future research, the VQA4ITM dataset will be made publicly accessible.

Versatile recurrent neural network for wide types of video restoration

Video shooting of natural scenes often suffers from various serious degradation, such as motion blur, impact of atmospheric turbulence, random noise and resolution reduction, etc. Different from the maturity of image restoration research, video restoration is much more complicated so that it lacks effective general method. Here, we present a versatile recurrent neural network (VRNN) to handle wide types of video degradation and generate stable videos with ideal clarity. We complete the design of VRNN through deducing a general video restoration paradigm that reveals the importance of simultaneously utilizing past and future information for restoring current frame. Specifically, we propose a novel RNN cell in which hidden state flows in bidirections, enriching temporal information contained in the extracted features. Furthermore, a feature fusion module involves temporal and spatial attention processing is designed to refine features of neighbouring frames and help reconstruct current frame. Extensive experiments on well-known public datasets (including four different kinds of video restoration tasks, with a total of 35,666 videos and 515,774 frames) show that the proposed VRNN achieves 1–4 dB of PSNR increasing or several times less of computational complexity in all tasks against state-of-the-art methods, manifesting the versatile and efficient ability of proposed VRNN in wide types of video restoration.

Improved Video Steganography with Dual Cover Medium, DNA and燙omplex燜rames

The most valuable resource on the planet is no longer oil, but data. The transmission of this data securely over the internet is another challenge that comes with its ever-increasing value. In order to transmit sensitive information securely, researchers are combining robust cryptography and steganographic approaches. The objective of this research is to introduce a more secure method of video steganography by using Deoxyribonucleic acid (DNA) for embedding encrypted data and an intelligent frame selection algorithm to improve video imperceptibility. In the previous approach, DNA was used only for frame selection. If this DNA is compromised, then our frames with the hidden and unencrypted data will be exposed. Moreover the frame selected in this way were random frames, and no consideration was made to the contents of frames. Hiding data in this way introduces visible artifacts in video. In the proposed approach rather than using DNA for frame selection we have created a fake DNA out of our data and then embedded it in a video file on intelligently selected frames called the complex frames. Using chaotic maps and linear congruential generators, a unique pixel set is selected each time only from the identified complex frames, and encrypted data is embedded in these random locations. Experimental results demonstrate that the proposed technique shows minimum degradation of the stenographic video hence reducing the very first chances of visual surveillance. Further, the selection of complex frames for embedding and creation of a fake DNA as proposed in this research have higher peak signal-to-noise ratio (PSNR) and reduced mean squared error (MSE) values that indicate improved results. The proposed methodology has been implemented in Matlab.

Verification of delay time and image compression thresholds for telesurgery.

Telerobotic surgery relies on communication lines, causing delays, and video information requires pre-transmission compression. Such delays and video degradation will continue to be unavoidable making communication conditions verification essential. Understanding the network specification values required for telerobotic surgery entails determining acceptable levels of delay and degradation due to the video compression and restoration processes during surgery. The hinotori™ surgical robot from Medicaroid was used. Eight surgeons, skilled in robotic surgery, performed gastrectomy or rectal resection on pigs. Image compression (bitrate: 120, 60, 30, 20, 10 Mbps) was random, changing encoder settings during surgery, and delay times (30, 50, 100, 150 milliseconds) were pseudo-randomly inserted, changing emulator settings. Acceptable video levels were evaluated. Subjective evaluations by surgeons and evaluators regarding image degradation and operability, and objective evaluations of image degradation and operability were given five-point ratings. Regarding delay time, 30 and 50 millisecond periods garnered average ratings of 3.6 and 4.0, respectively, signifying that surgery was feasible. However, at 100 and 150 millisecond, average ratings were 2.9 and 2.3, respectively, indicating surgery was not feasible for the most part in these cases. The average rating for image compression was 4.0 or higher for bitrates of 20, 30, 60, and 120 Mbps, suggesting that surgery is possible even at bitrates as low as 10Mbps, with an average rating of 4.0. In remote robotic surgery using the hinotori™, image compression and delay time are largely acceptable, so surgery can be safely performed.

Impact of the suboptimal communication network environment on telerobotic surgery performance and surgeon fatigue.

Remote surgery social implementation necessitates achieving low latency and highly reliable video/operation signal transmission over economical commercial networks. However, with commercial lines, communication bandwidth often fluctuates with network congestion and interference from narrowband lines acting as bottlenecks. Therefore, verifying the effects on surgical performance and surgeon fatigue when communication lines dip below required bandwidths are important. To clarify the communication bandwidth environment effects on image transmission and operability when bandwidth is lower than surgical robot requirements, and to determine surgeon fatigue levels in suboptimal environments. Employing a newly developed surgical robot, a commercial IP-VPN line connected two hospitals 150 km apart. Thirteen surgical residents remotely performed a defined suturing procedure at 1-Gbps to 3-Mbps bandwidths. Communication delay, packet loss, time-to-task completion, forceps-movement distance, video degradation, and robot operability were evaluated before and after bandwidth changes. The Piper Fatigue Score-12 (PFS-12) was used to measure fatigue associated with surgeon performance. Roundtrip communication time for both 1-Gbps and 3-Mbps lines averaged 4 ms. Video transmission delay from camera to monitor was comparable, at 92 ms. Surgical robot signal transmission rate averaged 5.2 Mbps, so changing to 1-Gbps-3-Mbps lines resulted in significant packet loss. Surgeons perceived significant roughness, image distortion, diplopia, and degradation of 3D images (p = 0.009), but not changes in delay time or maneuverability. All surgeons could complete tasks, but objective measurement of task-completion time and forceps-travel distance were significantly prolonged (p = 0.013, p = 0,041). Additionally, PFS-12 showed post-procedure fatigue increase at both 1-Gbps and 3-Mbps. Fatigue increase was significant at 3-Mbps (p = 0.041). In remote surgery environments with less than the optimal bandwidth, even when delay time and operability are equivalent, reduced surgical performance occurs from video degradation from packet loss. This may cause increased surgeon fatigue.

Selective Video Encryption Using the Cross Coupling of One-dimensional Logistic Maps

H.264 videos have been the most shared type of video format in recent times and hence its security is a major issue. The techniques presented in the recent times incur complex computations. The major research objective is to design an efficient Chaotic Selective Video Encryption (CSVE) technique which can result in a better visual degradation of the encrypted video with less computational complexity. In the proposed work, in order to secure the H.264 videos, two one-dimensional logistic maps are cross coupled in the chaotic encryption technique which uses a lookup table for data conversion. The technique is evaluated using different performance metrics like Peak Signal to Noise Ratio (PSNR), entropy, statistical analysis etc along with the traditional logistic map. The work is compared with some recent techniques in terms of PSNR and was found out that the proposed technique has better encryption effect.

Semantic Information Oriented No-Reference Video Quality Assessment

In this letter, a method called Semantic Information Oriented No-Reference (SIONR) video quality assessment model is developed, which can effectively represent quality degradation of video by taking the variations of semantic information into consideration. Specially, temporal variations of the semantic features between adjacent frames are calculated to consider the inconsistency of the static semantic information. Moreover, low-level features are also applied as a supplementary to take distortions related to local details into consideration. Experimental results demonstrate that our proposed method obtains competitive performance compared with state-of-the-art methods in the two databases. Also, our model achieves good generalization capability. The code is available at: https://github.com/lorenzowu/SIONR .

Auditory and Visual Integration for Emotion Recognition and Compensation for Degraded Signals are Preserved With Age.

Since emotion recognition involves integration of the visual and auditory signals, it is likely that sensory impairments worsen emotion recognition. In emotion recognition, young adults can compensate for unimodal sensory degradations if the other modality is intact. However, most sensory impairments occur in the elderly population and it is unknown whether older adults are similarly capable of compensating for signal degradations. As a step towards studying potential effects of real sensory impairments, this study examined how degraded signals affect emotion recognition in older adults with normal hearing and vision. The degradations were designed to approximate some aspects of sensory impairments. Besides emotion recognition accuracy, we recorded eye movements to capture perceptual strategies for emotion recognition. Overall, older adults were as good as younger adults at integrating auditory and visual information and at compensating for degraded signals. However, accuracy was lower overall for older adults, indicating that aging leads to a general decrease in emotion recognition. In addition to decreased accuracy, older adults showed smaller adaptations of perceptual strategies in response to video degradations. Concluding, this study showed that emotion recognition declines with age, but that integration and compensation abilities are retained. In addition, we speculate that the reduced ability of older adults to adapt their perceptual strategies may be related to the increased time it takes them to direct their attention to scene aspects that are relatively far away from fixation.

Degraded visual and auditory input individually impair audiovisual emotion recognition from speech-like stimuli, but no evidence for an exacerbated effect from combined degradation

Emotion recognition requires optimal integration of the multisensory signals from vision and hearing. A sensory loss in either or both modalities can lead to changes in integration and related perceptual strategies. To investigate potential acute effects of combined impairments due to sensory information loss only, we degraded the visual and auditory information in audiovisual video-recordings, and presented these to a group of healthy young volunteers. These degradations intended to approximate some aspects of vision and hearing impairment in simulation. Other aspects, related to advanced age, potential health issues, but also long-term adaptation and cognitive compensation strategies, were not included in the simulations. Besides accuracy of emotion recognition, eye movements were recorded to capture perceptual strategies. Our data show that emotion recognition performance decreases when degraded visual and auditory information are presented in isolation, but simultaneously degrading both modalities does not exacerbate these isolated effects. Moreover, degrading the visual information strongly impacts recognition performance and on viewing behavior. In contrast, degrading auditory information alongside normal or degraded video had little (additional) effect on performance or gaze. Nevertheless, our results hold promise for visually impaired individuals, because the addition of any audio to any video greatly facilitates performance, even though adding audio does not completely compensate for the negative effects of video degradation. Additionally, observers modified their viewing behavior to degraded video in order to maximize their performance. Therefore, optimizing the hearing of visually impaired individuals and teaching them such optimized viewing behavior could be worthwhile endeavors for improving emotion recognition.

Expert and Crowdsourced Evaluation of Image Quality From a Novel Endoscopy Phone Light Adapter

To evaluate the image quality of cystourethroscopy using a novel 3D printed phone light adapter with subject expert and crowdsourced evaluators. A simple 3D printed light adaptor for a flexible cystoscopy was developed and made open source. Two videos were then recorded of a simulated cystourethroscopy, one using the novel adapter and the other using a traditional endoscopy light source. Expert evaluators (urology trainees and attendings) were then asked to evaluate the video quality using a double stimulus impairment scale. They were also asked to rate their level of confidence in using the novel adapter in clinical scenarios. Using Amazon's Mechanical Turk marketplace, 100 crowdsourced evaluators viewed the same videos and completed the same rating scale. The Mann-Whitney U test was then used to compare the expert and crowdsourced ratings. Expert and crowdsourced evaluators saw minimal degradation of video quality for the simulated urethroscopy (P= .66). However, while expert evaluators did identify degradation in the cystoscopy video, the crowdsourced evaluators did not (P=.012). 96% of the expert evaluators would either "often" or "always" use the novel adapter for difficult Foley placements and removal of ureteric stents. The novel light adapter caused minimal degradation in image quality for urethroscopy as compared to a traditional endoscopy light source, with vast majority of raters believing it would be adequate to perform common bedside cystoscopy procedures.

Diagonal XOR-Based FEC Method to Improve Burst-Loss Tolerance for 4K/8K UHDTV Transmission

Broadcasters are converting their live program production facilities to Internet Protocol (IP)-networks. The introduction of IP-networks has many possibilities such as cost reduction and remote production, and various efforts are underway. At present, the main issue is to consider the construction of equipment for 2K high definition television (HDTV), and it is expected that the introduction of higher speed 4K and 8K ultra-HDTV (UHDTV) will accelerate in the future. In video transmissions using high-speed signals, video degradation due to packet loss such as random loss and burst loss that occurs in the network-switch of a transmission path becomes a problem. In the case of 4K and 8K UHDTV transmissions, since the number of packets per unit time increases, this problem becomes more serious. Therefore, it is indispensable to apply quality compensation technology to achieve high-quality video transmission. Forward error correction (FEC) using erasure codes is a key compensation method for recovering lost packets at the receiving side in unidirectional transmission. In this paper, we propose an exclusive OR-based FEC method with improved burst loss tolerance. Using this method, we have developed a transmission device for uncompressed 8K UHDTV signals over IP-networks using 100-Gb/s Ethernet packets. Experimental and simulation results clarified that the proposed method can improve the number of recoverable packets at the time of burst loss to twice as many as the conventional Pro-MPEG FEC method.

UnB-AV: An Audio-Visual Database for Multimedia Quality Research

In this paper we present the UnB-AV database, which is a database of audio-visual sequences and quality scores aimed at multimedia quality research. The database contains a total of 140 source content, with a diverse semantic content, both in terms of the video and audio components. It also contains 2,320 test sequences with audio and video degradations, along with the corresponding quality and content subjective scores. The subjective scores were collected by performing 3 different psycho-physical experiments using the Immersive Methodology. The three experiments have been presented individually in previous studies. In the first experiment, only the video component of the audio-visual sequences were degraded with compression (H.264 and H.265) and transmission (packet-loss and frame freezing) distortions. In the second experiment, only the audio component of the audio-visual sequences were degraded with common audio distortions (clip, echo, chop, and background noise). Finally, in the third experiment the audio and video degradations were combined to degrade both audio and video components. The UnB-AV database is available for download from the site of the Laboratory of Digital Signal Processing of the University of Brasilia and The Consumer Digital Video Library (CDVL).

A novel method for copyright protection of digital videos using SWEA and ZPA technique

Digital copyright protection has become an effective way to prove the ownership and protect the multimedia contents from illegal use and unauthorized users. In order to prove the ownership of a video certain security program is embedded in a video and one of the ways of ensuring the ownership of a video is embedding the watermark in a video. In this paper, a new approach to digital video copyright protection, SWEA (Split watermark embedding algorithm) with Zero Padding Algorithm (ZPA) is proposed. With the help of this algorithm, it is hard to know the original pattern of watermark because of SWEA and minimizing the perceptual degradation of watermarked video because of ZPA. Here we are using ‘db1’ wavelet domain for embedding the watermark in the LL sub-band of the original identical frame (I-frame), based on the energy of high-frequency sub-band in an adaptive manner. SCD (Scene changed detection) is used to find out the identical frame (I-frame). The proposed algorithm has undergone various attacks, such as compression, uniform noise, Gaussian noise frame repetition and frame averaging attacks. The proposed algorithm, sustain all the above attacks and offers improved performance compared with the other methods from the literature.

A fine tuned tracking of vehicles under different video degradations

Estimating the tracking efficacy of vehicles in traffic videos is one of the most desirable analysis specially in the presence of a challenging weather conditions. In this paper, a fine-tuning Kalman filter based tracking system has been proposed so that it will work robustly on the traffic videos. Such tracking efficacy has been tested and tuned by integrating a system that calculates the distance between adjacent vehicles as a case study. The integrated system could provide some sort of traffic warning system according to the allowable traffic safety standards. Analysis has been utilized in two phases; phase (1): by changing the performance indices of Kalman filter parameters (initial estimation error, motion noise, and measurement noise). We have measured both average number of assigned tracks and processing time of interest in order to acquire best tuning decision. From observations, changing values of initial estimation error has no effect on the performance of the tracking efficacy however increasing both motion noise and measurement noise has an adverse impact on the tracking performance. Phase (2) by applying the integrated system on a degraded version of a captured urban traffic video to measure performance of the tracking procedure according to the existence of salt and pepper, Gaussian, and Speckle video degradations. Such video disturbances could perform an evaluation for some sort of challenging weather conditions (e.g., rain, fog, and reduced light conditions). It is obviously that average number of assigned tracks has been degraded in the presence of video disturbance with respect to percentage of occurrence and the appropriate statistical features (mean, and variance) of such degradation. Twelve different types of filtering mask have been applied in order to measure average number of assigned tracks (correct predictions) after the cleaning process. We have measured the deviation between both the no noise and the with noise traffic video to study effect of each filter mask within each noise type of video disturbance. Such deviation measurements introduce a decision making criteria for best tuning that increases the efficacy of the vehicular tracking.

Viewer-Aware Intelligent Efficient Mobile Video Embedded Memory

Embedded memory is a critical component in today’s mobile video processing systems, increasingly dominating power consumption and shortening battery life of mobile devices. Traditional hardware-level power optimization techniques usually come with significant implementation overhead to solve the memory failure problem during low-voltage operations. This paper presents a novel mobile video memory to exploit the power saving opportunities provided by a viewer experience under environmental visual interference. The viewing contexts, in particular the ambient luminance, significantly influence the quality of the viewer experience, and in the context with higher luminance levels, mobile users have higher tolerance to the video degradation. Accordingly, the memory failures can be introduced adaptively to achieve power savings without influencing the viewer experience. To meet the silicon area constraint in mobile devices, a simple but an efficient hardware implementation scheme is developed to minimize area overhead. The experimental results based on a 45-nm CMOS technology show that, as compared with the conventional memory design, the proposed technique can achieve up to 48% power savings with good perceivable quality and negligible implementation overhead.

Using modified background subtraction for detecting vehicles in Videos

In this paper, a comparison study has been introduced between the traditional foreground detector (background subtraction technique) and a modified background subtraction-based detector (empty frame subtraction technique). A case study for such analysis has been introduced for estimating the average vehicular speed and the level of crowdedness in three test traffic videos with five different indices; frame rate, resolution, number of frames, duration, and extension). The proposed modification in the background subtraction detector strategy aims to reduce vehicle detection processing time which increase vehicle tracking efficacy. In addition, video degradations (salt and pepper noise, Gaussian noise, and speckle noise) have been applied in both traditional and modified background subtraction. Results have reflected an obvious decrease in the processing time by almost 40% than the traditional background detector.

An information hiding method based on context-based adaptive variable length coding

Current video information hiding algorithms based on H.264 cannot meet the requirement of real-time transmission and get compromise between embedding strength and the changing amplitude of bits stream. To solve these problems, a real-time video information hiding method in CAVLC for H.264/AVC is proposed, which combines information hiding with CAVLC process, and embeds the secret information in the number of trailing coefficients of 4 × 4 macroblocks. Aiming at alleviating video degradation and the embedded secret in real-time transmission, the method is time-efficient and robust. Experimental results show that the embedding process cannot bring detectable video degradation, and only cause a small change in the length of the video stream. Moreover, the embedded secret information can be recovered correctly when being transferred under different RTP lost channels. And the secret information can be extracted directly from the encoded stream without resorting to the original video.

An information hiding method based on context-based adaptive variable length coding

Current video information hiding algorithms based on H.264 cannot meet the requirement of real-time transmission and get compromise between embedding strength and the changing amplitude of bits stream. To solve these problems, a real-time video information hiding method in CAVLC for H.264/AVC is proposed, which combines information hiding with CAVLC process, and embeds the secret information in the number of trailing coefficients of 4 × 4 macroblocks. Aiming at alleviating video degradation and the embedded secret in real-time transmission, the method is time-efficient and robust. Experimental results show that the embedding process cannot bring detectable video degradation, and only cause a small change in the length of the video stream. Moreover, the embedded secret information can be recovered correctly when being transferred under different RTP lost channels. And the secret information can be extracted directly from the encoded stream without resorting to the original video.

An adaptive HVS based video watermarking scheme for multiple watermarks using BAM neural networks and fuzzy inference system

An efficient reversible adaptive video watermarking scheme for multiple watermarks based on Bi-directional Associative Memory (BAM) Neural Networks and Fuzzy Inference System namely, Multi-BAM-FUZ scheme is proposed in this paper. The main goal of this paper is to design a robust video watermarking system which facilitates secure video transmission over a communication channel by maintaining a trade-off among imperceptibility, robustness and watermark capacity or payload. The BAM neural network supports creation of weight matrix (formed out of multiple images) and this matrix is embedded into the DWT uncorrelated mid frequency coefficients of all the components (Y, Cb, Cr) of every frames of the video with varying embedding strength ‘α’. This adaptive embedding strength is generated using the Fuzzy Inference System which takes HVS characteristics such as luminance, texture and edge of each frame as an input in the DWT transform. The simulations performed on various test videos demonstrate that the proposed Multi-BAM-FUZ not only outperforms other existing methods with respect to various video degradation processes, but also maintains a satisfactory image quality, robustness and payload. It is noted that, the implementation of the novel adaptive process enhances the visual quality of about 60.97dB in terms of PSNR and 0.9998 in terms of SSIM, robustness of about nearly 1.0000 and 0.9999 in terms of Normalized Cross Correlation (NCC) value and Bit Correction Rate (BCR) respectively against various attacks. Moreover, the proposed scheme facilitates high level of payload without affecting the imperceptibility and robustness level.