Low-quality Frames Research Articles

Reconstruction flow recurrent network for compressed video quality enhancement

We present a reconstruction flow for the task of compressed video quality enhancement (VQE). Compressed videos often suffer from various coding artifacts, such as blocking and blurring, especially under low bit-rate. VQE aims to suppress these artifacts to improve the visual quality. Frame similarity can be utilized to enhance low-quality frames given their neighboring high-quality frames, for which motion estimation becomes important. Previous approaches often calculate optical flow for the motion compensation. On the other hand, video coding contains a rich set of block motion vectors, forming a coding flow, which may or may not correspond to the scene motion, but to places that deliver the minimum compression error. In contrast, such a valuable coding flow has always been ignored in VQE previously. In this work, we combine these two motion sources into a new flow, namely reconstruction flow, for the purpose of high-quality VQE. Specifically, we estimate optical flows from RGB frames and extract coding flows from coding streams, which are then merged by a fusion module to generate reconstruction flow. Besides, our network is built upon a recurrent network to utilize global temporal information. The deep features are warped according to the reconstruction flow and fed into the subsequent reconstruction module with spatial-variant kernel attention. Our method is evaluated on the leading MFQE2.0 dataset, which demonstrates superior performances when compared to the existing state-of-the-art methods.

A Keyframe Extraction Approach for 3D Videogrammetry Based on Baseline Constraints

In this paper, we propose a novel approach for the extraction of high-quality frames to enhance the fidelity of videogrammetry by combining fuzzy frames removal and baseline constraints. We first implement a gradient-based mutual information method to filter out low-quality frames while preserving the integrity of the videos. After frame pose estimation, the geometric properties of the baseline are constrained by three aspects to extract the keyframes: quality of relative orientation, baseline direction, and base to distance ratio. The three-dimensional model is then reconstructed based on these extracted keyframes. Experimental results demonstrate that our approach maintains a strong robustness throughout the aerial triangulation, leading to high levels of reconstruction precision across diverse video scenarios. Compared to other methods, this paper improves the reconstruction accuracy by more than 0.2 mm while simultaneously maintaining the completeness.

VRT: A Video Restoration Transformer.

Video restoration aims to restore high-quality frames from low-quality frames. Different from single image restoration, video restoration generally requires to utilize temporal information from multiple adjacent but usually misaligned video frames. Existing deep methods generally tackle with this by exploiting a sliding window strategy or a recurrent architecture, which are restricted by frame-by-frame restoration. In this paper, we propose a Video Restoration Transformer (VRT) with parallel frame prediction ability. More specifically, VRT is composed of multiple scales, each of which consists of two kinds of modules: temporal reciprocal self attention (TRSA) and parallel warping. TRSA divides the video into small clips, on which reciprocal attention is applied for joint motion estimation, feature alignment and feature fusion, while self attention is used for feature extraction. To enable cross-clip interactions, the video sequence is shifted for every other layer. Besides, parallel warping is used to further fuse information from neighboring frames by parallel feature warping. Experimental results on five tasks, including video super-resolution, video deblurring, video denoising, video frame interpolation and space-time video super-resolution, demonstrate that VRT outperforms the state-of-the-art methods by large margins (up to 2.16dB) on fourteen benchmark datasets. The codes are available at https://github.com/JingyunLiang/VRT.

Spatiotemporal Deep Learning-Based Cine Loop Quality Filter for Handheld Point-of-Care Echocardiography.

The reliability of automated image interpretation of point-of-care (POC) echocardiography scans depends on the quality of the acquired ultrasound data. This work reports on the development and validation of spatiotemporal deep learning models to assess the suitability of input ultrasound cine loops collected using a handheld echocardiography device for processing by an automated quantification algorithm (e.g. ejection fraction estimation). POC echocardiograms (n=885 DICOM cine loops from 175 patients) from two sites were collected using a handheld ultrasound device and annotated for image quality at the frame-level. Attributes of high-quality frames for left ventricular (LV) quantification included a temporally-stable LV, reasonable coverage of LV borders, and good contrast between the borders and chamber. Attributes of low-quality frames included temporal instability of the LV and/or imaging artifacts (e.g., lack of contrast, haze, reverberation, acoustic shadowing). Three different neural network architectures were investigated - (a) frame-level convolutional neural network (CNN) which operates on individual echo frames (VectorCNN), (b) single-stream sequence-level CNN which operates on a sequence of echo frames (VectorCNN+LSTM) and (c) two-stream sequence-level CNNs which operate on a sequence of echo and optical flow frames (VectorCNN+LSTM+Average, VectorCNN+LSTM+MinMax, and VectorCNN+LSTM+ConvPool). Evaluation on a sequestered test dataset containing 76 DICOM cine loops with 16,914 frames showed that VectorCNN+LSTM can effectively utilize both spatial and temporal information to regress the quality of an input frame (accuracy: 0.925, sensitivity = 0.860, specificity = 0.952), compared to the frame-level VectorCNN that only utilizes spatial information in that frame (accuracy: 0.903, sensitivity = 0.791, specificity = 0.949). Furthermore, an independent sample t-test indicated that the cine loops classified to be of adequate quality by the VectorCNN+LSTM model had a statistically significant lower bias in the automatically estimated EF (mean bias = - 3.73 ± 7.46 %, versus a clinically obtained reference EF) compared to the loops classified as inadequate (mean bias = -15.92 ± 12.17 %) (p = 0.007). Thus, cine loop stratification using the proposed spatiotemporal CNN model improves the reliability of automated point-of-care echocardiography image interpretation.

Video Compression using Neural Weight Step and Huffman Coding Techniques

Video Compression using Neural Weight Step and Huffman Coding Techniques

Dynamic method to optimize memory space requirement in real-time video surveillance using convolution neural network

ABSTRACT Real-time video surveillance is one of the most effective ways to observe crime, mischief, and violence. But, most of the recent surveillance system consumes huge memory space to store the video. This article proposed an advanced dynamic video surveillance strategy to utilize minimum memory space with the finest detection of suspicious moving objects. In the proposed system, a convolution neural network (CNN) classifier and frame resolution switching is used to detect the movement of objects with appropriate frame resolution. A high-definition (HD) frame can be recorded dynamically when any movement of a suspicious object is detected. The system records low-quality frames, which are less important. Finally, the improved gradient-based histogram equalization technique is applied to all frames to obtain enhanced suspicious frames. Several real-time imperial tests are conducted and it is observed that the proposed system detects suspicious objects with 98.25% accuracy. Besides, the system consumes 80% less memory storage.

Fluorescein videoangiography data analysis protocol for mapping retinal vascular permeability in humans.

Fluorescein video angiographies (FVAs) are a diagnostic tool for eye diseases, such as diabetic retinopathy (DR). Currently, kinetic tracer model methods based on indicator-dilutions theory use FVAs to extract biomarkers (e.g., volumetric blood flow and retinal vascular permeability) via pixel mapping using two-step non-linear least square fitting. Prior to biomarker extraction, the FVAs must attain optimal quality. The objective of this research is to create a program to remove all frames experiencing signal drops (causes include blinking, squinting, and head movement). 15 FVAs (6 healthy control subjects, 6 diabetes mellitus no DR (DMnoDR) subjects, and 3 mild non-proliferative DR (NPDR) subjects) were analyzed for low quality frames. The average signal of each frame was analyzed as top, middle, and bottom thirds. The frame with maximum average signal up to the final frame of a created "Gold Standard" was compared with the raw AVI's frame with maximum average signal and subsequent frames. All frames before maximum average signal and any remaining frames were compared with the previous good-quality raw frame to determine if the frame of interest was of good quality. All remaining frames were subsequently re-evaluated and flagged if they had a local minimum prominence of 10% of the maximum average signal. The flagged frames', as well as former and subsequent frames', quality were subjectively determined. The AVI quality was subsequently tested via pre-DTKM processing and biomarker extraction via DTKM methods. Results displayed that the semi-automated frame removal process provides sufficient quality AVIs.

P123 Development of an end-to-end Mayo endoscopic score machine learning algorithm for use in clinical trials of ulcerative colitis

Abstract Background Scoring colonoscopy videos of UC patients requires a high level of expertise, but even among trained expert readers there are disagreements. In clinical trials, this can negatively impact both subject selection and assessment of treatment response. Consistency among central readers might be improved by algorithms that automatically process videos to identify salient features and estimate the level of disease activity using established scoring methods. Methods We propose an end-to-end system using machine learning (ML) models to process colonoscopy videos, both shortening the time required for a human expert reader to score a video by filtering out non-informative parts and supplying a second-opinion of Mayo Endoscopic Score (MES) on each informative frame and the full video. Our dataset included videos from UC patients with a representative range of disease activity acquired at a Ukrainian hospital (n=505) and an Israeli hospital (n=227). Video annotation was performed by 12 reviewers, trained and supervised by an expert gastroenterologist (DM) with >10 years central reading experience. Out of almost 3 million frames over ~34% were classified by human reviewers as non-informative (e.g., out of focus, motion blurring, stool). The remaining informative frames were further annotated by the reviewers as containing ulcers (6%) and hence classified as MES=3, erosions (22%) as MES=2, loss of vascularity or erythema (23%) as MES= 1, or none of the above (49%) classified as MES=0. Informative frames were split to 80% training set, 10% validation set and 10% test set. The data set included low quality frames on which the reviewers could still estimate the MES. Models consisting of image processing algorithms, Convolutional Neural Networks (CNN), CNN+Long Short Term Memory (LSTM) model and classical ML classifiers were trained to filter out non-informative frames and score independently each frame and the full video for MES. Results Model performance was evaluated using Cohen’s Quadratic Weighted Kappa (QWK) with 95% Confidence Interval [95% CI]..Model agreement with humans is shown in the middle column. Inter-reviewer agreement based on a subset of ‘clear frames’ from various videos of the test set is shown in the last column. Conclusion We developed a full end-to-end pipeline for processing colonoscopy videos that estimated MES with accuracy comparable to human-human agreement. Such a model has the potential to make human readers more efficient by highlighting frames with relevant pathology. It can also aid inter-reader agreement by providing a second-opinion MES. Future work will expand model training and testing with new data sources and explore paradigms to combine the model with human readers to improve clinical trial central reading.

Polyp detection on video colonoscopy using a hybrid 2D/3D CNN.

Colonoscopy is the gold standard for early diagnosis and pre-emptive treatment of colorectal cancer by detecting and removing colonic polyps. Deep learning approaches to polyp detection have shown potential for enhancing polyp detection rates. However, the majority of these systems are developed and evaluated on static images from colonoscopies, whilst in clinical practice the treatment is performed on a real-time video feed. Non-curated video data remains a challenge, as it contains low-quality frames when compared to still, selected images often obtained from diagnostic records. Nevertheless, it also embeds temporal information that can be exploited to increase predictions stability. A hybrid 2D/3D convolutional neural network architecture for polyp segmentation is presented in this paper. The network is used to improve polyp detection by encompassing spatial and temporal correlation of the predictions while preserving real-time detections. Extensive experiments show that the hybrid method outperforms a 2D baseline. The proposed architecture is validated on videos from 46 patients and on the publicly available SUN polyp database. A higher performance and increased generalisability indicate that real-world clinical implementations of automated polyp detection can benefit from the hybrid algorithm and the inclusion of temporal information.

Video-based Person re-identification with parallel correction and fusion of pedestrian area features.

Deep learning has provided powerful support for person re-identification (person re-id) over the years, and superior performance has been achieved by state-of-the-art. While under practical application scenarios such as public monitoring, the cameras' resolutions are usually 720p, the captured pedestrian areas tend to be closer to 128×64 small pixel size. Research on person re-id at 128×64 small pixel size is limited by less effective pixel information. The frame image qualities are degraded and inter-frame information complementation requires a more careful selection of beneficial frames. Meanwhile, there are various large differences in person images, such as misalignment and image noise, which are harder to distinguish from person information at the small size, and eliminating a specific sub-variance is still not robust enough. The Person Feature Correction and Fusion Network (FCFNet) proposed in this paper introduces three sub-modules, which strive to extract discriminate video-level features from the perspectives of "using complementary valid information between frames" and "correcting large variances of person features". The inter-frame attention mechanism is introduced through frame quality assessment, guiding informative features to dominate the fusion process and generating a preliminary frame quality score to filter low-quality frames. Two other feature correction modules are fitted to optimize the model's ability to perceive information from small-sized images. The experiments on four benchmark datasets confirm the effectiveness of FCFNet.

MFQE 2.0: A New Approach for Multi-Frame Quality Enhancement on Compressed Video.

The past few years have witnessed great success in applying deep learning to enhance the quality of compressed image/video. The existing approaches mainly focus on enhancing the quality of a single frame, not considering the similarity between consecutive frames. Since heavy fluctuation exists across compressed video frames as investigated in this paper, frame similarity can be utilized for quality enhancement of low-quality frames given their neighboring high-quality frames. This task is Multi-Frame Quality Enhancement (MFQE). Accordingly, this paper proposes an MFQE approach for compressed video, as the first attempt in this direction. In our approach, we first develop a Bidirectional Long Short-Term Memory (BiLSTM) based detector to locate Peak Quality Frames (PQFs) in compressed video. Then, a novel Multi-Frame Convolutional Neural Network (MF-CNN) is designed to enhance the quality of compressed video, in which the non-PQF and its nearest two PQFs are the input. In MF-CNN, motion between the non-PQF and PQFs is compensated by a motion compensation subnet. Subsequently, a quality enhancement subnet fuses the non-PQF and compensated PQFs, and then reduces the compression artifacts of the non-PQF. Also, PQF quality is enhanced in the same way. Finally, experiments validate the effectiveness and generalization ability of our MFQE approach in advancing the state-of-the-art quality enhancement of compressed video.

DT-SLAM: Dynamic Thresholding Based Corner Point Extraction in SLAM System

Visual localization estimation is highly depended on the quality of video frames or captured images. Estimation quality may be affected by the poor visibility, low background texture and overexposure. Low quality frames with blurred edges and poor contrast pose tremendous difficulties for corner point detection in SLAM impacting the overall accuracy of estimation. This paper introduces DT-SLAM, a dynamic self-adaptive threshold (DSAT) approach for ORB corner point extraction in FAST to improve SLAM’s localization performance. The proposed method replaces the existing static threshold-based ORB extraction approach, enabling improved performance in complex real-world scenes. In addition, this study introduces a threshold switching mechanism (TSM) to replace the existing SLAM’s frame-level and cell-level thresholds for corner point extraction. The proposed DT-SLAM approach is validated using the TUM RGB-D and EuRoC benchmark datasets for location tracking performances. The results indicate that the proposed DT-SLAM outperforms the current state-of-the-art ORB-SLAM3, especially in challenging environments.

An Automatic System for Unconstrained Video-Based Face Recognition

Although deep learning approaches have achieved performance surpassing humans for still image-based face recognition, unconstrained video-based face recognition is still a challenging task due to large volume of data to be processed and intra/inter-video variations on pose, illumination, occlusion, scene, blur, video quality, etc. In this work, we consider challenging scenarios for unconstrained video-based face recognition from multiple-shot videos and surveillance videos with low-quality frames. To handle these problems, we propose a robust and efficient system for unconstrained video-based face recognition, which is composed of modules for face/fiducial detection, face association, and face recognition. First, we use multi-scale single-shot face detectors to efficiently localize faces in videos. The detected faces are then grouped through carefully designed face association methods, especially for multi-shot videos. Finally, the faces are recognized by the proposed face matcher based on an unsupervised subspace learning approach and a subspace-to-subspace similarity metric. Extensive experiments on challenging video datasets, such as Multiple Biometric Grand Challenge (MBGC), Face and Ocular Challenge Series (FOCS), IARPA Janus Surveillance Video Benchmark (IJB-S) for low-quality surveillance videos and IARPA JANUS Benchmark B (IJB-B) for multiple-shot videos, demonstrate that the proposed system can accurately detect and associate faces from unconstrained videos and effectively learn robust and discriminative features for recognition.

Biprediction-Based Video Quality Enhancement via Learning.

Convolutional neural networks (CNNs)-based video quality enhancement generally employs optical flow for pixelwise motion estimation and compensation, followed by utilizing motion-compensated frames and jointly exploring the spatiotemporal correlation across frames to facilitate the enhancement. This method, called the optical-flow-based method (OPT), usually achieves high accuracy at the expense of high computational complexity. In this article, we develop a new framework, referred to as biprediction-based multiframe video enhancement (PMVE), to achieve a one-pass enhancement procedure. PMVE designs two networks, that is, the prediction network (Pred-net) and the frame-fusion network (FF-net), to implement the two steps of synthesization and fusion, respectively. Specifically, the Pred-net leverages frame pairs to synthesize the so-called virtual frames (VFs) for those low-quality frames (LFs) through biprediction. Afterward, the slowly fused FF-net takes the VFs as the input to extract the correlation across the VFs and the related LFs, to obtain an enhanced version of those LFs. Such a framework allows PMVE to leverage the cross-correlation between successive frames for enhancement, hence capable of achieving high accuracy performance. Meanwhile, PMVE effectively avoids the explicit operations of motion estimation and compensation, hence greatly reducing the complexity compared to OPT. The experimental results demonstrate that the peak signal-to-noise ratio (PSNR) performance of PMVE is fully on par with that of OPT while its computational complexity is only 1% of OPT. Compared with other state-of-the-art methods in the literature, PMVE is also confirmed to achieve superior performance in both objective quality and visual quality at a reasonable complexity level. For instance, PMVE can surpass its best counterpart method by up to 0.42 dB in PSNR.

SAR Analysis for Real Video Super- Resolution Improvement

This paper presents a simultaneous autoregressive (SAR) analysis method to describe the unknown signal-to-noise ratio (SNR) and the texture feature of low-quality real video frames when the ground-truth images are not available. The real video images degraded by the factors, such as electronic noise, oversaturated pixels, motion blur, and compression artifacts, often result in poor motion registration estimation, which makes the performance of the existing video super-resolution (VSR) algorithms lower than expected. It is hard to estimate the SNR of the low-quality real frames without any prior knowledge. To solve this problem, we made a connection between SAR hyperparameters and the SNR of real images. The relationship expression of them was given in this paper. Using the proposed method, well-registered low-quality real video frames can be selected to decrease the root mean squared error (RMSE) of motion estimation of video frames for VSR reconstruction improvement. The anomalous low-quality frame images whose SAR hyperparameters values are inconsistent with others will be considered for removal. Synthetic experiments were designed to illustrate how the SAR hyperparameters values vary with the variation of synthetic parameters. In order to better illustrate the effectiveness of the proposed method, real low-quality videos captured under different conditions were tested under VSR reconstruction experiments. The VSR reconstruction results show that the results obtained using SAR prior analysis have sharper edges and fewer ringing artifacts than the original results. It indicates that the proposed method is helpful to obtain better results of motion registration estimation for low-quality real video images.

Video-Based Human Activity Recognition for Elderly Using Convolutional Neural Network

A typical healthcare application for elderly people involves monitoring daily activities and providing them with assistance. Automatic analysis and classification of an image by the system is difficult compared to human vision. Several challenging problems for activity recognition from the surveillance video involving the complexity of the scene analysis under observations from irregular lighting and low-quality frames. In this article, the authors system use machine learning algorithms to improve the accuracy of activity recognition. Their system presents a convolutional neural network (CNN), a machine learning algorithm being used for image classification. This system aims to recognize and assist human activities for elderly people using input surveillance videos. The RGB image in the dataset used for training purposes which requires more computational power for classification of the image. By using the CNN network for image classification, the authors obtain a 79.94% accuracy in the experimental part which shows their model obtains good accuracy for image classification when compared with other pre-trained models.

Learning Discriminative Aggregation Network for Video-Based Face Recognition and Person Re-identification

In this paper, we propose a discriminative aggregation network method for video-based face recognition and person re-identification, which aims to integrate information from video frames for feature representation effectively and efficiently. Unlike existing video aggregation methods, our method aggregates raw video frames directly instead of the features obtained by complex processing. By combining the idea of metric learning and adversarial learning, we learn an aggregation network to generate more discriminative images compared to the raw input frames. Our framework reduces the number of image frames per video to be processed and significantly speeds up the recognition procedure. Furthermore, low-quality frames containing misleading information can be well filtered and denoised during the aggregation procedure, which makes our method more robust and discriminative. Experimental results on several widely used datasets show that our method can generate discriminative images from video clips and improve the overall recognition performance in both the speed and the accuracy for video-based face recognition and person re-identification.

CoLiTec software - detection of the near-zero apparent motion

AbstractIn this article we described CoLiTec software for full automated frames processing. CoLiTec software allows processing the Big Data of observation results as well as processing of data that is continuously formed during observation. The scope of solving tasks includes frames brightness equalization, moving objects detection, astrometry, photometry, etc. Along with the high efficiency of Big Data processing CoLiTec software also ensures high accuracy of data measurements. A comparative analysis of the functional characteristics and positional accuracy was performed between CoLiTec and Astrometrica software. The benefits of CoLiTec used with wide field and low quality frames were observed. The efficiency of the CoLiTec software was proved by about 700.000 observations and over 1.500 preliminary discoveries.

A pixel-based outlier-free motion estimation algorithm for scalable video quality enhancement

Scalable video quality enhancement refers to the process of enhancing low quality frames using high quality ones in scalable video bitstreams with time-varying qualities. A key problem in the enhancement is how to search for correspondence between high quality and low quality frames. Previous algorithms usually use block-based motion estimation to search for correspondences. Such an approach can hardly estimate scale and rotation transforms and always introduces outliers to the motion estimation results. In this paper, we propose a pixel-based outlier-free motion estimation algorithm to solve this problem. In our algorithm, the motion vector for each pixel is calculated with respect to estimate translation, scale, and rotation transforms. The motion relationships between neighboring pixels are considered via the Markov random field model to improve the motion estimation accuracy. Outliers are detected and avoided by taking both blocking effects and matching percentage in scale-invariant feature transform field into consideration. Experiments are conducted in two scenarios that exhibit spatial scalability and quality scalability, respectively. Experimental results demonstrate that, in comparison with previous algorithms, the proposed algorithm achieves better correspondence and avoids the simultaneous introduction of outliers, especially for videos with scale and rotation transforms.

Image Quality Assessment and Outliers Filtering in an Image-Based Animal Supervision System

This paper presents a probabilistic framework for the image quality assessment (QA), and filtering of outliers, in an image-based animal supervision system (asup). The proposed framework recognizes asup's imperfect frames in two stages. The first stage deals with the similarity analysis of the same-class distributions. The objective of this stage is to maximize the separability measures by defining a set of similarity indicators (SI) under the condition that the number of permissible values for them is restricted to be relatively low. The second stage, namely faulty frame recognition (FFR), deals with asup's QA training and real-time quality assessment (RTQS). In RTQS, decisions are made based on a real-time quality assessment mechanism such that the majority of the defected frames are removed from the consecutive sub routines that calculate the movements. The underlying approach consists of a set of SI indexes employed in a simple Bayesian inference model. The results confirm that a significant amount of defected frames can be efficiently classified by this approach. The performance of the proposed technique is demonstrated by the classification on a cross-validation set of mixed high and low quality frames. The classification shows a true positive rate of 88.6% while the false negative rate is only about 2.5%.