Reference Video Sequences Research Articles

Perceptual Quality Assessment of NeRF and Neural View Synthesis Methods for Front‐Facing Views

AbstractNeural view synthesis (NVS) is one of the most successful techniques for synthesizing free viewpoint videos, capable of achieving high fidelity from only a sparse set of captured images. This success has led to many variants of the techniques, each evaluated on a set of test views typically using image quality metrics such as PSNR, SSIM, or LPIPS. There has been a lack of research on how NVS methods perform with respect to perceived video quality. We present the first study on perceptual evaluation of NVS and NeRF variants. For this study, we collected two datasets of scenes captured in a controlled lab environment as well as in‐the‐wild. In contrast to existing datasets, these scenes come with reference video sequences, allowing us to test for temporal artifacts and subtle distortions that are easily overlooked when viewing only static images. We measured the quality of videos synthesized by several NVS methods in a well‐controlled perceptual quality assessment experiment as well as with many existing state‐of‐the‐art image/video quality metrics. We present a detailed analysis of the results and recommendations for dataset and metric selection for NVS evaluation.

UHD Database Focus on Smart Cities and Smart Transport

“Smart city” refers to a modern solution to organizing a city’s services, using cloud technologies to collect and evaluate large amounts of data, including data from camera systems. Smart city management covers several areas that can be implemented separately, but only their combination can realize the overall desired smart city function. One of the core areas of smart city automation is smart city transport. Transportation is a crucial system in any city, and this is why it needs to be monitored. The primary objective of this publication is to generate top-notch 4K UHD video sequences that are solely dedicated to showcasing smart cities and their transportation systems. The resulting comprehensive database will be made accessible to all professionals in the field, who can utilize it for extensive research purposes. Additionally, all the reference video sequences will be transcoded into various quality settings by altering critical parameters like the resolution, compression standard, and bit rate. The ultimate aim is to determine the best combination of video parameters and their respective settings based on the measured values. This in-depth evaluation will ensure that each video sequence is of the highest quality and provides an unparalleled experience for the service providers offering the service. The video sequences captured will be analyzed for quality assessments in smart cities or smart transport technologies. The database will also include objective and subjective ratings, along with information about the dynamics determined by spatial and temporal information. This will enable a comparison of the subjective evaluation of a selected sample of our respondents with the work of other researchers, who may evaluate it with a different sample of evaluators. The assumption of our future research is to predict the subjective quality based on the type of sequence determined by its dynamicity.

Feature‐based no‐reference video quality assessment using Extra Trees

With the emergence of social networks and improvements in the internet speed, the video data has become an ever-increasing portion of the global internet traffic. Besides the content, the quality of a video sequence is an important issue at the user end which is often affected by various factors such as compression. Therefore, monitoring the quality is crucial for the video content and service providers. A simple monitoring approach is to compare the raw video content (uncompressed) with the received data at the receiver. In most practical scenarios, however, the reference video sequence is not available. Consequently, it is desirable to have a general reference-less method for assessing the perceived quality of any given video sequence. In this paper, a no-reference video quality assessment technique based on video features is proposed. In particular, a long list of video features (21 sets of features, each consisting of 1 to 216 features) is considered and all possible combinations ( 2 21 − 1 ${2^{21}-1}$ ) for training an Extra Trees regressor is examined. This choice of the regressor is wisely selected and is observed to perform better than other common regressors. The results reveal that the top 20 performing feature subsets all outperform the existing feature-based assessment methods in terms of the Pearson linear correlation coefficient (PLCC) or the Spearman rank order correlation coefficient (SROCC). Specially, the best performing regressor achieves PLCC = 0.786 ${\text{ PLCC}=0.786}$ on the test data over the KonVid-1k dataset. It is believed that the results of the comprehensive comparison could be potentially useful for other feature-based video-related problems. The source codes of the implementations are publicly available.

A Selective Video Encryption Scheme Based on Coding Characteristics

The protection of video data has become a hot topic of research. Researchers have proposed a series of coding algorithms to ensure the safe and efficient transmission of video information. We propose an encryption scheme that can protect video information with higher security by combining the video coding algorithm with encryption algorithm. The H.264/AVC encoding algorithm encodes the video into multiple slices, and the slices are independent of each other. With this feature, we encrypt each slice while using the cipher feedback (CFB) mode of the advanced encryption standard (AES) with the dynamic key. The key is generated by the pseudo-random number generator (PRNG) and updated in real time. The encryption scheme goes through three phases: constructing plaintext, encrypting plaintext, and replacing the original bitstream. In our scheme, we encrypt the code stream after encoding, so it does not affect the coding efficiency. The purpose of the CFB mode while using the AES encryption algorithm is to maintain the exact same bit rate and produce a format compatible bitstream. This paper proposes a new four-dimensional (4-D) hyperchaotic algorithm to protect data privacy in order to further improve the security of video encryption. Symmetric encryption requires that the same key is used for encryption and decoding. In this paper, the symmetry method is used to protect the privacy of video data due to the large amount of video encrypted data. In the experiment, we evaluated the proposed algorithm while using different reference video sequences containing motion, texture, and objects.

Using automated video analysis to study fish escapement through escape panels in active fishing gears: Application to the effect of net colour

Quantification of the escapement rate of unwanted fish through a selective device is usually based on catch comparison. This study proposes a new, time efficient method to automatically compare two selective devices by automated counting of fish escapements through each selective device based on video sequences. First, sea trials were conducted to record video sequences of fish escaping a white and black square mesh panel. Then, all of the underwater sequences were automatically analysed by a computer vision software for automated object detection and tracking. Finally, the algorithm was assessed using 150 min of video sequences analysed by humans. We observed that the variability in escapements rate between all the observers on reference video sequences could reach 5%. As the difference in escapements rate between the algorithm and the observers was lower than the variability between observers, the automated approach was validated. The software detected a significant difference in fish escapement rate according to the net colour in the camera field of view: 60% of all fish escaped through the white panel. Our results suggest that net colour influences the escape rates of fish. The colour of the selective device should therefore be investigated further with the aim of increasing their efficiency. Further development of the software could be done to identify species and size of the fish and assess the effectiveness of a selective device by species and size.

Effective Video Frame Acquisition for Image Stitching

We present an effective video frame (including reference frame and key frames) acquisition method for image stitching. The method simultaneously analyzes different types of factors, namely, the video-level stability, image-level stability, and content scale stability, to take advantage of their complementary strengths. We model the three factors with three modules that are learned from an analysis of the shooting process. The video stabilization module (VSM) selects a stable segment, while the shooting distance module (SDM) obtains a similar content scale. They collaborate during the reference video sequence so that they can benefit from each other. Then, the image quality module (IQM) obtains a reference frame from the above sequence by choosing high-quality images. Finally, to obtain the key frame set, the SDM and IQM are again used to continuously filter the overlapping video sequences formed by the reference frame or the latest key frame. In particular, a comprehensive dataset containing a variety of challenges and scenarios is introduced. We have conducted an extensive set of experiments on this dataset. The results confirm the effectiveness of each module and their collaboration; our method outperforms current state-of-the-art methods.

Subjective and Objective Study of the Relation Between 3D and 2D Views Based on Depth and Bitrate

The tremendous growth in 3D (stereo) imaging and display technologies has led to stereoscopic content (video and image) becoming increasingly popular. However, both the subjective and the objective evaluation of stereoscopic video content has not kept pace with the rapid growth of the content. Further, the availability of standard stereoscopic video databases is also quite limited. In this work, we attempt to alleviate these shortcomings. We present a stereoscopic video database and its subjective evaluation. We have created a database containing a set of 144 distorted videos. We limit our attention to H.264 compression artifacts. The distorted videos were generated using 6 uncompressed pristine videos of left and right views originally created by Ecole Polytechnique Federal De Lausanne (EPFL)[1]. The reference video sequences contain a good combination of texture, motion, depth information and we divided these videos into 2 groups based on depth information. Further, 19 subjects participated in the subjective assessment task. Based on the subjective study, we have formulated a conditional relation between the 2D and stereoscopic subjective scores as a function of compression rate and depth range. We have also evaluated the performance of popular 2D and 3D image/video quality assessment (I/VQA) algorithms on our database.

Performance Evaluation of Visual Tracking Algorithms on Video Sequences With Quality Degradation

Recently, there are lots of visual tracking algorithms proposed to improve the performance of object tracking in video sequences with various real conditions, such as severe occlusion, complicated background, fast motion, and so on. In real visual tracking systems, there are various quality degradation occurring during video acquisition, transmission, and processing. However, most existing studies focus on improving the accuracy of visual tracking while ignoring the performance of tracking algorithms on video sequences with certain quality degradation. In this paper, we investigate the performance evaluation of existing visual tracking algorithms on video sequences with quality degradation. A quality-degraded video database for visual tracking (QDVD-VT), including the reference video sequences and their corresponding distorted versions, is constructed as the benchmarking for robustness analysis of visual tracking algorithms. Based on the constructed QDVD-VT, we propose a method for robustness measurement of visual tracking (RMVT) algorithms by accuracy rate and performance stability. The performance of ten existing visual tracking algorithms is evaluated by the proposed RMVT based on the built QDVD-VT. We provide the detailed analysis and discussion on the robustness analysis of different visual tracking algorithms on video sequences with quality degradation from different distortion types. To visualize the robustness of visual tracking algorithms well, we design a robustness pentagon to show the accuracy rate and performance stability of visual tracking algorithms. Our initial investigation shows that it is still challenging for effective object tracking for existing visual tracking algorithms on video sequences with quality degradation. There is much room for the performance improvement of existing tracking algorithms on video sequences with quality degradation in real applications.

Reduced-reference video quality assessment using a static video pattern

A reduced-reference video quality assessment (VQA) method was proposed by using structural similarity (SSIM) index as a tool to extract features from both the original and the target video sequences, using a reference video pattern. The method is suitable for monitoring the video quality in real time and across the service provision chain. The performance of the proposed method was evaluated using a large experimental set of reference and nonreference video sequences and achieves an accuracy higher than 2.56% in comparison to SSIM. Additionally, comparison to subjectively evaluated scores of Laboratory for Image and Video Engineering video quality dataset, based on difference mean opinion scores, shows that the performance of the proposed method is within the range of the full reference VQA methods.

Estimation of Degraded Video Quality of Mobile H.264/AVC Video Streaming

The problem of estimation of subjective video quality of distorted video sequences for mobile video streaming is addressed. Widely spreading mobile communication systems and increasing data transmission rates expand variety of multimedia services. One of such services is video straming. So it is important to asses quality of this service. Consumers of video streaming are humans, and quality assessment must account human perception characteristics. Most of the well-known video quality estimation methods, such as Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM), Video Quality Metric (VQM), etc. are based on the presence of full or reduced reference video sequence. However, in order to estimate user experienced video quality, methods with no reference must be employed. Such existing methods as quality metrics usually use bit-error rate that has low correlation with by human perceived video quality. More advanced methods usually require too much processing power that cannot be obtained in handled mobile devices. It is shown that video quality estimation method with no reference and low computation complexity could be developed basing on H.264/AVC video stream structure. Such method is suitable for implementation in mobile devices with low processing power. Ill. 4, bibl. 11 (in English; summaries in English, Russian and Lithuanian).

Network centric QoS performance evaluation of IPTV transmission quality over VANETs

Internet Protocol Television (IPTV) traffic streaming over vehicular ad hoc networks (VANETs) challenge could be distinguished by the frequent VANETs topology change, paired with real-time streaming of IPTV traffic which requires high bandwidth and strict performance service quality. For VANETs to deliver better quality IPTV traffic, the network must satisfy the compelling Quality of Service (QoS) requirement of real-time traffic streaming, such as jitter, bandwidth, delay, and loss. There are numerous metrics defined in the literature for evaluating multimedia streaming traffic QoS such as: video quality metric, peak-signal-to-noise-ratio, moving picture quality metric and many more. However, these metrics relied upon the objective approach of quality evaluation that requires the original signal as reference, and cannot isolate the impact of network impairments on the video quality. Therefore, fail to provide any mapping between the network QoS parameters and the respective deteriorated quality of the multimedia traffic. Similarly, such procedures are not practically applicable to VANETs whose network characteristics make it practically impossible to access the reference video sequence. Hence, in this paper, we conduct an experiment to determine the feasibility of delivering a qualitative IPTV service over VANETs. We derived analytical model to quantify the IPTV QoS influencing parameters, where we establish relationships between the variables of each parameter and their impacts on the IPTV traffic QoS. Through extensive experiment, we evaluate the IPTV transmission QoS parameters, to assure a priority for handling bandwidth allocation, delay and loss control to a negligible level.

Selecting Method of Sending Block Images Based on Time-Domain Frequency Estimation for Low Delay Live Video Streaming System

To realize low delay live video streaming system over lossy network, each captured frame must be divided into block images, and these images must be encoded, packetized, and sent selectively. To improve the quality of reproduced frames at the receiver side, the selecting method of the block images must be sophisticated. This paper proposed an optimal selecting method of block images. The method consists of three phases based on the formulated two optimization problems. Firstly, a frequency analysis method is applied to the recent luminance value for each block area. Secondly, sendable block ratio is optimally distributed among block areas based on the result of frequency analysis to satisfy the available bandwidth condition. Finally, the optimal timings of sending are planned for each block area to minimize the error of reproduced image under the condition of assigned sendable block ratio. Some experiments were conducted to evaluate the proposed method using three frequently used reference video sequences. The results of comparison with the conventional method shows that the proposed method improved the mean squared error (MSE) of reproduced image up to 64%. Another results shows that the robustness of the proposed method against packet loss can be changed by modifying the parameter of the optimization problem.

An attack invariant scheme for content-based video copy detection

Due to the advancement in the field of multimedia technology and communication, it has become easier to access, store, and edit video data. Easy manipulation of video data and its rapid distribution have made content-based video copy detection (CBVCD) an active area of research. In a CBVCD system, reference video sequence and query sequence are compared to detect whether the query video sequence is a copy of reference video sequence. Thus, the generation of fingerprint of a video sequence and sequence matching technique are the core tasks of such system. In order to evade such detection process, a copied version may undergo different kinds of transformations like photometric and post-production attack. So the detection system must be robust enough against such attacks. In this work, fingerprint is generated from the sub-bands of wavelet decomposed intensity image and localized intensity gradient histograms of low sub-band. The fingerprint thus obtained reflects considerable discriminating capability and robustness against the attacks. Furthermore, to cope up with the attacks, we have adopted simple pre-processing technique, which enhances the robustness of the system further. A robust sequence matching technique based on multivariate Wald–Wolfowitz test is proposed here. An experiment has been carried out with a database consisting of distinct 642 shots and 1,485 query sequences representing different attacks. Proposed methodology achieves high copy detection rate (99.39 %) and very low false alarm rate (0.14 %) and performs better than other spatio-temporal measure based systems.

Digital Video Quality Metric Based on Watermarking Technique with Geffe Generator

This paper presents a proposed approach for assessing video quality without reference and makes use of a data hiding technique to embed a fragile mark into video frame by using Discreet Cosine Transform (DCT) and quantization. The fragile mark is random watermark generator using stream cipher based on Linear Feedback Shift Register (LFSR) and using Geffe generator to give a balanced distribution of zeros and ones in its output. The frame format consists of three color components, Red (R), Green (G) and Blue (B) for individual pixel, the watermark data should be hidden in Red (R) color channel to ensure the best recovery of the watermark . At the receiver, the mark is extracted from decoded video without any original reference video sequences. After extracted watermark, quality measure of the video is obtained by computing the degradation of the extracted mark. The results of this experiment indicate identical values of the Normalized Cross-correlation (NC) for three categories of real quality that have been proposed (good, low, bad) with the perceived quality. The experiment shows the error of the proposed system is 7% while correct ratio is 93% , the salt and pepper noise gives good results without errors while the Dust& Scratches noise gives highest error in the system. Results shown that the proposed algorithm provides a good estimation for the video quality when adding noise.

SB-DRED: a novel congestion control algorithm for H.264 video streaming over 802.11e

The IEEE 802.11e EDCA provides differentiated service for multimedia transmission via in-station multiple access category (AC) queues and virtual collision handling for wireless channel access contention. However, the FIFO nature of AC_VI (AC for Video) limits its performance. To provide further service differentiation among video packets, this paper presents a novel cross-layer design called E-DVFI+SB-DRED, i.e. equal-population-grouping of dispersive-video-frame-importance (E-DVFI) for a precise video importance scheme in the Application layer and segmented-and-bounded dispersive-random-early-detection (SB-DRED) in the MAC layer for a hyperfine video packet mapping algorithm among AC_VI and its lower-priority ACs. PSNR-based performance evaluations and comparisons including the effects of video input rate to AC_VI and cross-traffic via the other ACs have been considered over reference video sequences, and the results show that the proposed cross-layer design consistently outperforms the existing ones, including EDCA, I/P+AM, and our previous study DVFI+DRED.

IPTV에서 평가주기 변화가 주관적 영상 화질 평가에 미치는 영향 분석

There have been considerable standardization works to measure subjective video quality. However, these standardization methods are based on the situation where they have original or reference video sequences, and measure the video quality by looking the sequence during short time duration, e.g. 10 seconds. Since people`s TV watching time durations in IPTV services are very long (e.g. more than 30 minutes), the quality measurements by the standards are not practical. In this paper, we analyze the effect of video quality rating period to MOS (Mean Opinion Score), and suggest a proper rating duration of subjective video quality for IPTV services.

Video-based reconstruction of animatable human characters

We present a new performance capture approach that incorporates a physically-based cloth model to reconstruct a rigged fully-animatable virtual double of a real person in loose apparel from multi-view video recordings. Our algorithm only requires a minimum of manual interaction. Without the use of optical markers in the scene, our algorithm first reconstructs skeleton motion and detailed time-varying surface geometry of a real person from a reference video sequence. These captured reference performance data are then analyzed to automatically identify non-rigidly deforming pieces of apparel on the animated geometry. For each piece of apparel, parameters of a physically-based real-time cloth simulation model are estimated, and surface geometry of occluded body regions is approximated. The reconstructed character model comprises a skeleton-based representation for the actual body parts and a physically-based simulation model for the apparel. In contrast to previous performance capture methods, we can now also create new real-time animations of actors captured in general apparel.