Compressed Frames Research Articles

ACORN+: Adaptive Compression-Reconstruction for Device-Cloud Collaboration Video Services

With the improvement of edge-based autonomous systems such as mobile Industrial IoT(IIoT) networks, edge devices can capture and upload videos with increasing bitrates. Massive edge-computing end nodes are eager for adequate multimedia data to satisfy the requirements of real-time video services. However, existing encoding standards for video services in Web 2.0 are specifically designed for something other than IoT video streaming. We have improved our Adaptive Compression-Reconstruction framework ACORN to obtain ACORN+, based on compressed sensing and recent advances in deep learning. At end nodes, we compress multiple sequential video frames into a single frame to reduce video volume. Given that multiple kinds of intelligent tasks are expected to be finished on the device side, we also designed a device-cloud collaboration scheme where deep learning-based algorithms can be executed on both the device and server sides. Experiments reveal that video analytics can be conducted on compressed frames. Taking action recognition as a device-cloud collaboration use case, we find ACORN+ obtains more than 3x speedup on compressed frames. The reconstruction algorithm in ACORN+ is with 1- 4 dB improvements. Moreover, the encoding time cost and the encoded video volume are reduced by more than 4x under the ACORN+ framework. 1

Optimal object stitching and video condensation-aided improved YOLOv5-oriented anomaly object detection framework

Recent advances in intelligent surveillance to boost public security have increased interest in anomaly location and identification among computer vision researchers in congested situations. Unrestricted video monitoring is required due to its extensive use in private and public spheres. Particularly, research on finding anomalies in surveillance videos has drawn much interest. Video condensation is an efficient method for quickly scanning and retrieving lengthy surveillance films. Reordering the video events in the spatial or temporal domain seeks to condense large video sequences into their equivalent compact video representation. Additionally, if a delay occurs in anomaly detection, it leads to time consumption for rectifying it. Therefore, the core aim of this task is to reduce the time consumption of detecting the instance of video anomaly detection faster by introducing the optimal object stitching through the deep learning-based video condensation concept. The steps include video condensation and deep learning-based anomaly detection. Initially, the necessary video for anomaly detection is gathered from the datasets. The gathered video is converted into frames and is used for object detection in each frame using Improved YOLOv5 (I-YOLOv5). Then, the optimal frame selection is performed using the hybrid Fitness of Wild Geese and Path Finder (FWGPF) for stitching the detected objects into the selected optimal frames, thus compressing the relevant frames. In the second phase, the condensed (compressed) frames are utilized for determining the anomaly frames with the support of Parameter Tuning in Dilated MobileNet (PTDM), and further, the anomaly objects are detected on the classified condensed frames based on the I-YOLOv5. Here, the tuning of constraint is done in the anomaly detection phase by the same hybrid FWGPF to reduce the time consumption. The experimental analysis of the developed model is performed with baseline algorithms and detection techniques to confirm the developed model’s efficiency.

Uncertainty of digital image correlation under video compression and DSP optimization.

The storage and transmission of videos at high spatial resolution remain a great challenge in image-based optical techniques. The uncertainty of digital image correlation (DIC) was assessed following speckle video compression under High Efficiency Video Coding (HEVC/H.265). First, the evaluation criterion for the DIC accuracy affected by compression was provided. The stability of H.265 video compression in DIC was studied considering different compressed frames under different target quantization parameters (QPs) and compression ratios (CRs). The deformation uncertainty of the DIC itself as affected by H.265 video compression was further investigated through uniform translation and non-uniform sinusoidal deformation performance. Moreover, the optimized digital speckle pattern (DSP) was re-evaluated considering video compression-induced uncertainty. DSPs with parameters of different diameters and randomness were compressed using various QPs and CRs. In addition, DSP evaluation was performed under both translation and non-homogeneous deformation conditions. The feasibility of the re-optimized DSP under H.265 video compression was validated using a defective bending beam, and DSP videos with a speckle size of 8 pixels reached a high CR within an acceptable margin of error.

EPN2EOS Data Transfer System

EPN2EOS Data Transfer System

Video Compression using Neural Weight Step and Huffman Coding Techniques

Video Compression using Neural Weight Step and Huffman Coding Techniques

Video Compression Artifact Reduction by Fusing Motion Compensation and Global Context in a Swin-CNN Based Parallel Architecture

Video Compression Artifact Reduction aims to reduce the artifacts caused by video compression algorithms and improve the quality of compressed video frames. The critical challenge in this task is to make use of the redundant high-quality information in compressed frames for compensation as much as possible. Two important possible compensations: Motion compensation and global context, are not comprehensively considered in previous works, leading to inferior results. The key idea of this paper is to fuse the motion compensation and global context together to gain more compensation information to improve the quality of compressed videos. Here, we propose a novel Spatio-Temporal Compensation Fusion (STCF) framework with the Parallel Swin-CNN Fusion (PSCF) block, which can simultaneously learn and merge the motion compensation and global context to reduce the video compression artifacts. Specifically, a temporal self-attention strategy based on shifted windows is developed to capture the global context in an efficient way, for which we use the Swin transformer layer in the PSCF block. Moreover, an additional Ada-CNN layer is applied in the PSCF block to extract the motion compensation. Experimental results demonstrate that our proposed STCF framework outperforms the state-of-the-art methods up to 0.23dB (27% improvement) on the MFQEv2 dataset.

Improving Compressed Video Using Single Lightweight Model with Temporal Fusion Module.

Video compression algorithms are commonly used to reduce the number of bits required to represent a video with a high compression ratio. However, this can result in the loss of content details and visual artifacts that affect the overall quality of the video. We propose a learning-based restoration method to address this issue, which can handle varying degrees of compression artifacts with a single model by predicting the difference between the original and compressed video frames to restore video quality. To achieve this, we adopted a recursive neural network model with dilated convolution, which increases the receptive field of the model while keeping the number of parameters low, making it suitable for deployment on a variety of hardware devices. We also designed a temporal fusion module and integrated the color channels into the objective function. This enables the model to analyze temporal correlation and repair chromaticity artifacts. Despite handling color channels, and unlike other methods that have to train a different model for each quantization parameter (QP), the number of parameters in our lightweight model is kept to only about 269 k, requiring only about one-twelfth of the parameters used by other methods. Our model applied to the HEVC test model (HM) improves the compressed video quality by an average of 0.18 dB of BD-PSNR and -5.06% of BD-BR.

Textural and Directional Information Based Offset In-Loop Filtering in AVS3

In this paper, we propose a novel low-complexity in-loop filtering approach named textural and directional information based offset (TDIO) for the emerging video coding standard AVS3 (the third generation of Audio Video-coding Standard). Different from conventional offset-based filtering methods which partially use contextual samples, the key contribution of TDIO is that it fully utilizes the textural and edge directional features of each sample to comprehensively determine which type of texture characteristics each sample belongs to. The corresponding offsets are generated and signaled to decoder such that sample-level distortion is reduced by correcting the quantization errors. Specifically, the multi-directionality and sample-intensity pattern based classifiers are first proposed to extract the directional and textural features, respectively. The classification results are obtained by incorporating these features, and the optimal offset values for each class are derived based on rate-distortion optimization. Since sample-level offset signalling may cause heavy burden to the overhead of TDIO, we subsequently propose a filtering offset sharing mechanism based on historical information between available temporal-adjacent compressed frames. In addition, an iteration-based filter adaptation method is designed to improve the local adaptivity of TDIO for better compression efficiency. Experimental results show that the proposed TDIO achieves 0.64%, 1.29%, 1.86%, and 2.20% bit rate savings for all intra, random access, low delay B, and low delay P configurations, respectively. Moreover, TDIO is helpful to improve subjective quality by leveraging the fine-grained local texture characteristics. It can be observed that the blurring and ringing artifacts could be significantly suppressed by using the proposed method, yielding higher subjective quality.

Paper 2: Performing rapid reviews

BackgroundHealth policy-makers must often make decisions in compressed time frames and with limited resources. Hence, rapid reviews have become a pragmatic alternative to comprehensive systematic reviews. However, it is important that rapid review methods remain rigorous to support good policy development and decisions. There is currently little evidence about which streamlined steps in a rapid review are less likely to introduce unacceptable levels of uncertainty while still producing a product that remains useful to policy-makers.MethodsThis paper summarizes current research describing commonly used methods and practices that are used to conduct rapid reviews and presents key considerations and options to guide methodological choices for a rapid review.ResultsThe most important step for a rapid review is for an experienced research team to have early and ongoing engagement with the people who have requested the review. A clear research protocol, derived from a needs assessment conducted with the requester, serves to focus the review, defines the scope of the rapid review, and guides all subsequent steps. Common recommendations for rapid review methods include tailoring the literature search in terms of databases, dates, and languages. Researchers can consider using a staged search to locate high-quality systematic reviews and then subsequently published primary studies. The approaches used for study screening and selection, data extraction, and risk-of-bias assessment should be tailored to the topic, researcher experience, and available resources. Many rapid reviews use a single reviewer for study selection, risk-of-bias assessment, or data abstraction, sometimes with partial or full verification by a second reviewer. Rapid reviews usually use a descriptive synthesis method rather than quantitative meta-analysis. Use of brief report templates and standardized production methods helps to speed final report publication.ConclusionsResearchers conducting rapid reviews need to make transparent methodological choices, informed by stakeholder input, to ensure that rapid reviews meet their intended purpose. Transparency is critical because it is unclear how or how much streamlined methods can bias the conclusions of reviews. There are not yet internationally accepted standards for conducting or reporting rapid reviews. Thus, this article proposes interim guidance for researchers who are increasingly employing these methods.

Flow incorporated neural network based lightweight video compression architecture

The sudden surge in the video transmission over internet motivated the exploration of more promising and potent video compression architectures. Though the frame prediction based hand designed techniques are performing well and widely used but the recent deep learning based researches in this domain provided further directions of pure deep learning based next generation codecs. As the bandwidth over the internet is varying, adaptive bit rate representation is more suitable for video quality adjustment in tune with bandwidth variation. The proposed architecture comprises of end to end trainable video compression network consisting of majorly three modules namely-motion extension network, flow autoencoder and frame autoencoder. Frame autoencoder generates the individual compressed frames, flow autoencoder is used for optical flow based motion compensation chore and next frame is predicted by the motion extension network. The network is designed and evaluated in incremental manner. The analysis of the outcomes demonstrates the promising performance of the network quantitatively and qualitatively. Moreover, the results reveal that inclusion of optical flow based motion compensation network to the MotionNet architecture has enhanced the performance.

Deep Learning-Based Object Tracking via Compressed Domain Residual Frames

The extensive rise of high-definition CCTV camera footage has stimulated both the data compression and the data analysis research fields. The increased awareness of citizens to the vulnerability of their private information, creates a third challenge for the video surveillance community that also has to encompass privacy protection. In this paper, we aim to tackle those needs by proposing a deep learning-based object tracking solution via compressed domain residual frames. The goal is to be able to provide a public and privacy-friendly image representation for data analysis. In this work, we explore a scenario where the tracking is achieved directly on a restricted part of the information extracted from the compressed domain. We utilize exclusively the residual frames already generated by the video compression codec to train and test our network. This very compact representation also acts as an information filter, which limits the amount of private information leakage in a video stream. We manage to show that using residual frames for deep learning-based object tracking can be just as effective as using classical decoded frames. More precisely, the use of residual frames is particularly beneficial in simple video surveillance scenarios with non-overlapping and continuous traffic.

Advances in Video Compression System Using Deep Neural Network: A Review and Case Studies

Significant advances in video compression systems have been made in the past several decades to satisfy the near-exponential growth of Internet-scale video traffic. From the application perspective, we have identified three major functional blocks, including preprocessing, coding, and postprocessing, which have been continuously investigated to maximize the end-user quality of experience (QoE) under a limited bit rate budget. Recently, artificial intelligence (AI)-powered techniques have shown great potential to further increase the efficiency of the aforementioned functional blocks, both individually and jointly. In this article, we review recent technical advances in video compression systems extensively, with an emphasis on deep neural network (DNN)-based approaches, and then present three comprehensive case studies. On preprocessing, we show a switchable texture-based video coding example that leverages DNN-based scene understanding to extract semantic areas for the improvement of a subsequent video coder. On coding, we present an end-to-end neural video coding framework that takes advantage of the stacked DNNs to efficiently and compactly code input raw videos via fully data-driven learning. On postprocessing, we demonstrate two neural adaptive filters to, respectively, facilitate the in-loop and postfiltering for the enhancement of compressed frames. Finally, a companion website hosting the contents developed in this work can be accessed publicly at https://purdueviper.github.io/dnn-coding/.

Mutual information regularized identity-aware facial expression recognition in compressed video

How to extract effective expression representations that invariant to the identity-specific attributes is a long-lasting problem for facial expression recognition (FER). Most of the previous methods process the RGB images of a sequence, while we argue that the off-the-shelf and valuable expression-related muscle movement is already embedded in the compression format. In this paper, we target to explore the inter-subject variations eliminated facial expression representation in the compressed video domain. In the up to two orders of magnitude compressed domain, we can explicitly infer the expression from the residual frames and possibly extract identity factors from the I frame with a pre-trained face recognition network. By enforcing the marginal independence of them, the expression feature is expected to be purer for the expression and be robust to identity shifts. Specifically, we propose a novel collaborative min-min game for mutual information (MI) minimization in latent space. We do not need the identity label or multiple expression samples from the same person for identity elimination. Moreover, when the apex frame is annotated in the dataset, the complementary constraint can be further added to regularize the feature-level game. In testing, only the compressed residual frames are required to achieve expression prediction. Our solution can achieve comparable or better performance than the recent decoded image-based methods on the typical FER benchmarks with about 3 times faster inference.

One-Class Double Compression Detection of Advanced Videos Based on Simple Gaussian Distribution Model

Passive video forensics has become an active topic in recent years. Generally, a pristine video obtained from surveillance cameras or other video recording devices is single compressed. At the same time, double lossy compression will certainly be introduced in tampered videos since it needs to go through recompression to perform tampering on a commonly used compressed video. In the video’s double compression, some traces are left due to the intrinsic effects of recompression. Traditional supervised learning is inefficient because it requires two classes (the pristine and the manipulated) that occur in the video to be exhaustively assigned labels. Actually, compared with pristine videos, manipulated videos with labels are more difficult to obtain. To address this problem, in this paper, one-class classification, which is often used for anomaly detection and only needs the target class, is introduced. We first treat all decompressed video frames as still images and extract subtractive pixel adjacency matrix (SPAM) steganalysis features to detect traces left in the double compression process. Then we adopt a Gaussian density-based one-class classifier since SPAM features extracted from pristine video frames approximately subject to the Gaussian distribution. Furthermore, we improve the robustness of the classifier by using ensemble strategy. Experimental results indicate that our proposed method exceeds other more complex one-class classification methods, and outperforms fully-supervised learning methods only by feeding features from single compressed video frames to our one-class classifier.

MFQE 2.0: A New Approach for Multi-Frame Quality Enhancement on Compressed Video.

The past few years have witnessed great success in applying deep learning to enhance the quality of compressed image/video. The existing approaches mainly focus on enhancing the quality of a single frame, not considering the similarity between consecutive frames. Since heavy fluctuation exists across compressed video frames as investigated in this paper, frame similarity can be utilized for quality enhancement of low-quality frames given their neighboring high-quality frames. This task is Multi-Frame Quality Enhancement (MFQE). Accordingly, this paper proposes an MFQE approach for compressed video, as the first attempt in this direction. In our approach, we first develop a Bidirectional Long Short-Term Memory (BiLSTM) based detector to locate Peak Quality Frames (PQFs) in compressed video. Then, a novel Multi-Frame Convolutional Neural Network (MF-CNN) is designed to enhance the quality of compressed video, in which the non-PQF and its nearest two PQFs are the input. In MF-CNN, motion between the non-PQF and PQFs is compensated by a motion compensation subnet. Subsequently, a quality enhancement subnet fuses the non-PQF and compensated PQFs, and then reduces the compression artifacts of the non-PQF. Also, PQF quality is enhanced in the same way. Finally, experiments validate the effectiveness and generalization ability of our MFQE approach in advancing the state-of-the-art quality enhancement of compressed video.

Slow scholarship for social work: A praxis of resistance and creativity

Expediency, efficiency, and rapid production within compressed time frames represent markers for research and scholarship within the neoliberal academe. Scholars who wish to resist these practices of knowledge production have articulated the need for Slow scholarship—a slower pace to make room for thinking, creativity, and useful knowledge. While these calls are important for drawing attention to the costs and problems of the neoliberal academy, many scholars have moved beyond “slow” as being uniquely referencing pace and duration, by calling for the different conceptualizations of time, space, and knowing. Guided by post-structural feminisms, we engaged in a research project that moved at the pace of trust in the integrity of our ideas and relationships. Our case study aimed to better understand the ways macro forces such as neoliberalism, criminalization and professionalization shape domestic violence work. This article discusses our praxis of Slow scholarship by showcasing four specific key markers of Slow scholarship in our research; time reimagined, a relational ontology, moving inside and towards complexity, and embodiment. We discuss how Slow scholarship complicates how we understand constructs of productivity and knowledge production, as well as map the ways Slow scholarship offers a praxis of resistance for generating power from the epistemic margins within social work and the neoliberal academy.

A recurrent video quality enhancement framework with multi-granularity frame-fusion and frame difference based attention

In recent years, deep learning has attracted substantial research attention for video restoration. Among the existing contributions, the single-frame based approaches purely rely on one reference frame and neglect the rest neighboring frames when enhancing a target frame. By contrast, the multi-frame based contributions exploit temporal information in a sliding window and the existing recurrent design only employ a single preceding enhanced frame. It is intuitive to exploit both multiple original neighboring frames and the preceding enhanced frames for video quality enhancement. In this paper, we propose a Recurrent video quality Enhancement framework with Multi-granularity frame-fusion and frame Difference based attention (REMD). Firstly, we devise a three-dimensional convolutional neural network based encoder-decoder fusion model, which fuses multiple frames in multi-granularity. Secondly, severe compression artifacts tend to emerge on the edges and textures of the compressed frames. We propose a frame difference based spatial attention method to intensify the edges and textures of motioning regions. Finally, a recurrent sliding window design is conceived for exploiting the temporal information in preceding enhanced frames and subsequent neighboring frames. Experiments demonstrate that our method achieves superior performance in comparison to the state-of-the-art contributions with substantially reduced spatial and computational complexity.

Low cost image compression algorithm with colour reproduction algorithm for transmitting video frame of capsule endoscopy using low power

Presented a low cost technique for transmitting data through radio frequency antenna using low power, which is the combination of low computational complexity image compression algorithm and simple colour reproduction algorithm. In wireless capsule endoscopy, radio frequency antenna consume more than 60% of power for transmitting data and transmitting all the RGB components consume more power. In the proposed scheme, the first compressed colour frame is transmitted followed by few number of compressed grey frames. Compression algorithm is made of simplified YUV colour space, differential pulse code modulation and Golomb Rice code. At decoder dictionary based colour reproduction algorithm is used to colorize the grey scale image using colour information of original colour image. The proposed scheme lead to saving of more than 50% power of the battery, which increases the battery life from 10 h to 18 h. This extra battery life can be used for adding extra features to capsule, which help in development of diagnosing and treatment capsules.

Offset-Based In-Loop Filtering With a Deep Network in HEVC

With the great flexibility and performance of deep learning technology, there have been many attempts to replace existing functions inside video codecs such as High-Efficiency Video Coding (HEVC) with deep-learning-based solutions. One of the most researched approaches is adopting a deep network as an image restoration filter to recover distorted compressed frames. In this paper, instead, we introduce a novel idea for using a deep network, in which it chooses and transmits the side information according to the type of errors and contents, inspired by the sample adaptive offset filter in HEVC. A part of the network computes the optimal offset values while another part estimates the type of error and contents simultaneously. The combination of two subnetworks can address the estimation of highly nonlinear and complicated errors compared to conventional deep- learning-based schemes. Experimental results show that the proposed system yields an average bit-rate saving of 4.2% and 2.8% for the low-delay P and random access modes, respectively, compared to the conventional HEVC. Moreover, the performance improvement is up to 6.3% and 3.9% for higher-resolution sequences.

DNN Based Moth Search Optimization for Video Forgery Detection

Recently, video tampering process becomes easier due to the rapid advancements in user-friendly editing software and multimedia technology (e.g., Mokey by Imagineer Systems, and Photoshop and Premiere by Adobe). This technologies may highly tamper the original videos, so that the audience gets mislead. Nowadays, MPEG-4 codec is included in a large proportions of video cameras and surveillance systems. Therefore the double compression detection process included as an initial step in the video forensic is receiving a high significance. In this paper, the double compression artifacts is detected by adopting the Markov based features, which identifies the interpolated original videos. The double compressed frames are then segmented by introducing an SLIC super pixel segmentation technique. Here, the feature extraction is performed by applying the scale information that is obtained from the multi-scale Gabor filters. The features of this Gabor scale accurately extract the structural features and also reduce too much of redundancy. This extracted features are then provided to DNN (deep neural network) for forgery detection. In this video forensic process, DNN classifier is included for forgery detection. The CNN classifier is included in various existing forgery detection techniques. But, in our work we include DNN because it contains number of hidden layers which provide accurate results for this forgery detection process. To improve the DNN performance, Moth Search Optimization (MSO) algorithm is introduced in this forgery detection technique. Every nook and corner of this world we can able to find the surveillance cameras for security purpose. But, some fraudsters perform forgeries in this recorded videos for their own benefits. To identify this, a lot of forgery detection techniques are coming into existence. So in this work, we introduce the DNN based MSO to perform the forgery detection in videos.This implementation is processed in python simulation platform. The parametric evaluations are taken in terms of F1-Score, average accuracy, Precision, Recall and. Experimental results will provide improved performance in video forgery detection.