Residual Frame Research Articles

Structured residual sparsity for video compressive sensing reconstruction

Recent advancements in the residual sparsity strategy have garnered widespread attention in video compressive sensing (CS) reconstruction. However, most of the existing residual sparsity-based video CS reconstruction methods usually suffer from some limitations that lead to undesired visual artifacts. Firstly, these methods only rely on a patch sparsity scheme that is limited by their focus on the local structures of each video frame, neglecting the nonlocal self-similarity (NSS) property inherent to each video frame. Secondly, these methods concentrate on utilizing the NSS property of external reference frames for multi-hypothesis (MH) prediction while disregarding the internal NSS property of the current frame. In this paper, we propose a new structured residual sparsity (SRS) approach for video CS reconstruction, which jointly exploits the NSS properties of the current frame and its reference frames. Specifically, due to the unavailability of the original video frames, we first devise an effective intraframe CS (EICS) reconstruction method that leverages the internal NSS property of each frame. This approach enables us to obtain initial recovery frames, which then facilitate the execution of MH prediction. Following this, we generate a residual frame for the current frame by employing the MH prediction. Then, we propose a novel SRS model jointly using the NSS properties of the current frame and its reference frames to explore both the correlations of intraframe and interframe for reconstructing the current frame. Furthermore, for the sake of optimization feasibility, we develop an effective alternating direction method of multipliers (ADMM) algorithm to address the objective. Our experimental findings reveal that the proposed SRS not only yields superior quantitative results, but also uncovers finer details and causes fewer visual artifacts compared to many popular or state-of-the-art video CS reconstruction approaches.

Directional Residual Frame: Turns the motion information into a static RGB frame

Directional Residual Frame: Turns the motion information into a static RGB frame

Deep Parametric 3D Filters for Joint Video Denoising and Illumination Enhancement in Video Super Resolution

Despite the quality improvement brought by the recent methods, video super-resolution (SR) is still very challenging, especially for videos that are low-light and noisy. The current best solution is to subsequently employ best models of video SR, denoising, and illumination enhancement, but doing so often lowers the image quality, due to the inconsistency between the models. This paper presents a new parametric representation called the Deep Parametric 3D Filters (DP3DF), which incorporates local spatiotemporal information to enable simultaneous denoising, illumination enhancement, and SR efficiently in a single encoder-and-decoder network. Also, a dynamic residual frame is jointly learned with the DP3DF via a shared backbone to further boost the SR quality. We performed extensive experiments, including a large-scale user study, to show our method's effectiveness. Our method consistently surpasses the best state-of-the-art methods on all the challenging real datasets with top PSNR and user ratings, yet having a very fast run time. The code is available at https://github.com/xiaogang00/DP3DF.

A multi-behavior recognition method for macaques based on improved SlowFast network

Macaque is a common animal model in drug safety assessment. Its behavior reflects its health condition before and after drug administration, which can effectively reveal the side effects of drugs. At present, researchers usually rely on artificial methods to observe the behavior of macaque, which cannot achieve uninterrupted 24-hour monitoring. Therefore, it is urgent to develop a system to realize 24-hour observation and recognition of macaque behavior. In order to solve this problem, this paper constructs a video dataset containing nine kinds of macaque behaviors (MBVD-9), and proposes a network called Transformer-augmented SlowFast for macaque behavior recognition (TAS-MBR) based on this dataset. Specifically, the TAS-MBR network converts the red, green and blue (RGB) color mode frame input by its fast branches into residual frames on the basis of SlowFast network and introduces the Transformer module after the convolution operation to obtain sports information more effectively. The results show that the average classification accuracy of TAS-MBR network for macaque behavior is 94.53%, which is significantly improved compared with the original SlowFast network, proving the effectiveness and superiority of the proposed method in macaque behavior recognition. This work provides a new idea for the continuous observation and recognition of the behavior of macaque, and lays the technical foundation for the calculation of monkey behaviors before and after medication in drug safety evaluation.

Shake Table response of a dual system with inline friction damper

The purpose of this paper is to investigate the structural characteristics of a dual system moment frame and concentrically braced frame with a friction damper (CBFD). The experimental test involves the design, detailing, fabrication, and testing of a full-scale dual system frame on a uniaxial shake table subjected to 16 scaled ground motions of various intensities and amplitudes and eight artificially generated sinusoidal, sweep and step functions. The dual system utilized for this testing includes two ordinary moment-resisting frames and a single ordinary concentric braced frame with and without an inline friction damper. Overall, the friction damped braced system decreased the system acceleration by 25 percent at the frame deck. The friction-damped brace increased the damping of the dual system frame by 6.7 percent while maintaining story drift below 1.7 percent. The residual frame story drifts demonstrated that the ordinary moment-resisting frames participated as re-centering frames correcting any residual elongation in the friction damper. Overall, the inline friction damper assembled within the dual system frame demonstrated enhanced structural performance and resiliency.

Low Computational Coding-Efficient Distributed Video Coding: Adding a Decision Mode to Limit Channel Coding Load

Distributed video coding (DVC) is based on distributed source coding (DSC) concepts in which video statistics are used partially or completely at the decoder rather than the encoder. The rate-distortion (RD) performance of distributed video codecs substantially lags the conventional predictive video coding. Several techniques and methods are employed in DVC to overcome this performance gap and achieve high coding efficiency while maintaining low encoder computational complexity. However, it is still challenging to achieve coding efficiency and limit the computational complexity of the encoding and decoding process. The deployment of distributed residual video coding (DRVC) improves coding efficiency, but significant enhancements are still required to reduce these gaps. This paper proposes the QUAntized Transform ResIdual Decision (QUATRID) scheme that improves the coding efficiency by deploying the Quantized Transform Decision Mode (QUAM) at the encoder. The proposed QUATRID scheme's main contribution is a design and integration of a novel QUAM method into DRVC that effectively skips the zero quantized transform (QT) blocks, thus limiting the number of input bit planes to be channel encoded and consequently reducing both the channel encoding and decoding computational complexity. Moreover, an online correlation noise model (CNM) is specifically designed for the QUATRID scheme and implemented at its decoder. This online CNM improves the channel decoding process and contributes to the bit rate reduction. Finally, a methodology for the reconstruction of the residual frame (R^) is developed that utilizes the decision mode information passed by the encoder, decoded quantized bin, and transformed estimated residual frame. The Bjøntegaard delta analysis of experimental results shows that the QUATRID achieves better performance over the DISCOVER by attaining the PSNR between 0.06 dB and 0.32 dB and coding efficiency, which varies from 5.4 to 10.48 percent. In addition to this, results determine that for all types of motion videos, the proposed QUATRID scheme outperforms the DISCOVER in terms of reducing the number of input bit-planes to be channel encoded and the entire encoder's computational complexity. The number of bit plane reduction exceeds 97%, while the entire Wyner-Ziv encoder and channel coding computational complexity reduce more than nine-fold and 34-fold, respectively.

Optimized video compression with residual split attention and swin-block artifact contraction

Research in video compression has seen significant advancement in the last several years. However, the existing deep learning-based algorithms continue to be plagued by erroneous motion compression and ineffective motion compensation architectures, resulting in compression errors with a lower rate–distortion trade-off. To overcome these challenges, we present an end-to-end purely deep learning-based video compression method through a set of primary operations (e.g., motion estimation, motion compression, motion compensation, residual compression, and artifact contraction) differently. A deep residual attention split (DRAS) block is introduced for motion compression networks to pay more attention to certain image regions to create more effective features for the decoder while boosting the rate–distortion optimization (RDO) efficiency. A channel residual block (CRB) is proposed in motion compensation to yield a more accurate predicted frame, potentially improving the residual frame. To mitigate the compression errors, an artifact contraction module (ACM) by residual swin convolution UNet block is included in this model to improve the reconstruction quality. To improve the final frame, a buffer is added to fine-tune the previous reference frames. These modules combine with a loss function by assessing the trade-off and enhancing the decoded video quality. A comprehensive ablation study demonstrates the effectiveness of the proposed blocks and modules for video compression. Experimental results show the competitive performance of the proposed method on four benchmark datasets.

Attention-guided residual frame learning for video anomaly detection

Attention-guided residual frame learning for video anomaly detection

Video Summarization using Deep Convolutional Neural Networks and Mutual Probability-based K-Nearest Neighbour

ABSTRACT The video summarisation is an advanced mechanism for enabling users to handle and browse large videos in an effective manner. Various video summarisation methods are developed in recent days, in which handling of synchronisation and timing issues remain as the important challenge. The proposed video summarisation technique produces a short summary from the huge video stream. Initially, from an input database, the cricket videos containing number of frames are fed to keyframe extraction unit. Here, the keyframe extraction is done by the Euclidean distance and discrete cosine transform, and the best keyframes are selected based on the Euclidean distance. The residual frame is obtained by passing the input frames through deep convolutional neural network. Then, the similarity is calculated by Bhattacharyya distance. For video summarisation process, the optimal frameset is evaluated by matching residual keyframe with obtained keyframes. Here, input queries consisting of face object are subjected to object matching process, which is performed using the proposed mutual probability-based k-nearest neighbour (MP-KNN) to obtain relevant frames based on texture features. The performance of the proposed MP-KNN is superior based on precision, recall, and F-measure with values 0.963, 0.960, and 0.909, respectively.

TDCOSMO

In the context of gravitational lensing, the density profile of lensing galaxies is often considered to be perfectly elliptical. Potential angular structures are generally ignored, except to explain flux ratios of point-like sources (i.e. flux ratio anomalies). Surprisingly, the impact of azimuthal structures on extended images of the source has not been characterised, nor has its impact on the H0 inference. We address this task by creating mock images of a point source embedded in an extended source and lensed by an elliptical galaxy on which multipolar components are added to emulate boxy or discy isodensity contours. Modelling such images with a density profile free of angular structure allows us to explore the detectability of image deformation induced by the multipoles in the residual frame. Multipole deformations are almost always detectable for our highest signal-to-noise ratio (S/N) mock data. However, the detectability depends on the lens ellipticity and Einstein radius, on the S/N of the data, and on the specific lens modelling strategy. Multipoles also introduce small changes to the time-delays. We therefore quantify how undetected multipoles would impact H0 inference. When no multipoles are detected in the residuals, the impact on H0 for a given lens is in general less than a few km s−1 Mpc−1, but in the worst-case scenario, combining low S/N in the ring and large intrinsic boxyness or discyness, the bias on H0 can reach 10−12 km s−1 Mpc−1. If we now look at the inference on H0 from a population of lensing galaxies with a distribution of multipoles representative of what is found in the light profile of elliptical galaxies, we find a systematic bias on H0 of less than 1%. A comparison of our mock systems to the state-of-the-art time-delay lens sample studied by the H0LiCOW and TDCOSMO collaborations indicates that multipoles are currently unlikely to be a source of substantial systematic bias on the inferred value of H0 from time-delay lenses.

GaitSlice: A gait recognition model based on spatio-temporal slice features

Improving the performance of gait recognition under multiple camera views (i.e., cross-view gait recognition) and various conditions is urgent. From observation, we find that adjacent body parts are inter-related while walking, and each frame in a gait sequence possesses different degrees of semantic information. In this paper, we propose a novel model, GaitSlice, to analyze the human gait based on spatio-temporal slice features. Spatially, we design Slice Extraction Device (SED) to form top-down inter-related slice features. Temporally, we introduce Residual Frame Attention Mechanism (RFAM) to acquire and highlight the key frames. To better simulate reality, GaitSlice combines parallel RFAMs with inter-related slice features to focus on the features’ spatio-temporal information. We evaluate our model on CASIA-B and OU-MVLP gait datasets and compare it with six typical gait recognition models by using rank-1 accuracy. The results show that GaitSlice achieves high accuracy in gait recognition under cross-view and various walking conditions.

Deep Learning-Based Object Tracking via Compressed Domain Residual Frames

The extensive rise of high-definition CCTV camera footage has stimulated both the data compression and the data analysis research fields. The increased awareness of citizens to the vulnerability of their private information, creates a third challenge for the video surveillance community that also has to encompass privacy protection. In this paper, we aim to tackle those needs by proposing a deep learning-based object tracking solution via compressed domain residual frames. The goal is to be able to provide a public and privacy-friendly image representation for data analysis. In this work, we explore a scenario where the tracking is achieved directly on a restricted part of the information extracted from the compressed domain. We utilize exclusively the residual frames already generated by the video compression codec to train and test our network. This very compact representation also acts as an information filter, which limits the amount of private information leakage in a video stream. We manage to show that using residual frames for deep learning-based object tracking can be just as effective as using classical decoded frames. More precisely, the use of residual frames is particularly beneficial in simple video surveillance scenarios with non-overlapping and continuous traffic.

Experimental Validation of Detachable Links for Eccentrically Braced Frames

AbstractEccentrically braced frames (EBFs) subjected to seismic actions dissipate energy through yielding of link members. Other members of the framing are on the other hand designed to remain mainly elastic during a seismic event. The framing can be reused after an earthquake if the link members can be replaced. In line with this, the last decade has witnessed development of replaceable links. However, most of the proposed details are not well suited for replacement under residual drift conditions. Consequently, this paper presents details of an experimental validation of a detachable link concept where a splice connection is introduced at the mid‐length of the link to facilitate replacement under residual frame drift. The concept was studied by conducting six nearly full scale EBF tests under quasi‐static cyclic loading. Different splice details employing bolted connections with vertical slotted or site drilled holes and welded connections were developed. The link length ratio, type of mid‐splice connection and the amount of residual drift were considered as the variables of the testing program. The specimens were subjected to the loading protocol recommended in AISC Seismic Provisions for Structural Steel Buildings (AISC‐341). The experimental results suggested that the detachable links with the proposed connection details can satisfy the link rotation capacity limits of AISC‐341. The detachable links performed similar to conventional links while their replaceability under residual drift was demonstrated.

Mutual information regularized identity-aware facial expression recognition in compressed video

How to extract effective expression representations that invariant to the identity-specific attributes is a long-lasting problem for facial expression recognition (FER). Most of the previous methods process the RGB images of a sequence, while we argue that the off-the-shelf and valuable expression-related muscle movement is already embedded in the compression format. In this paper, we target to explore the inter-subject variations eliminated facial expression representation in the compressed video domain. In the up to two orders of magnitude compressed domain, we can explicitly infer the expression from the residual frames and possibly extract identity factors from the I frame with a pre-trained face recognition network. By enforcing the marginal independence of them, the expression feature is expected to be purer for the expression and be robust to identity shifts. Specifically, we propose a novel collaborative min-min game for mutual information (MI) minimization in latent space. We do not need the identity label or multiple expression samples from the same person for identity elimination. Moreover, when the apex frame is annotated in the dataset, the complementary constraint can be further added to regularize the feature-level game. In testing, only the compressed residual frames are required to achieve expression prediction. Our solution can achieve comparable or better performance than the recent decoded image-based methods on the typical FER benchmarks with about 3 times faster inference.

Mid-spliced end-plated replaceable links for eccentrically braced frames

Eccentrically braced frames (EBFs) can be re-used after a major seismic event by replacing the link members. Recent years have witnessed the development of numerous replaceable links. Among various details developed for this purpose, extended end-plated replaceable links are found to be the most efficient. The use of these links enables the engineer to minimize the size and the weight of the part to be replaced. In addition, the performance of end-plated links is similar to that of conventional links. Research reported to date has shown that end-plated links have disadvantages in terms of removal and replacement. Large axial forces can develop within the link member. Removal and replacement operations may require the use of hydraulic jacks to push the ends of the collector beams and the flame cutting of the links to gradually release the residual stresses and deformations. More important are the difficulties associated with link replacement under residual frame drifts. A novel detachable, replaceable link is proposed in this study which employs a splice connection at the mid-length of the link. The splice connection consists of channel sections welded to both parts of the replaceable link. The detail employed provides an erection tolerance which facilitates easy removal and enables replacement under residual frame drifts. Proof-of-concept testing of the proposed links was performed on 3 specimens where the type of force transfer in the splice connection was considered as the prime variable. All specimens failed at link rotation angles that were significantly higher than the link rotation angle required by AISC341 and demonstrated the potential of the proposed link concept. Complementary finite element parametric studies were conducted to validate the design procedure developed for the proposed replaceable link concept.

Rethinking Motion Representation: Residual Frames With 3D ConvNets.

Recently, 3D convolutional networks yield good performance in action recognition. However, an optical flow stream is still needed for motion representation to ensure better performance, whose cost is very high. In this paper, we propose a cheap but effective way to extract motion features from videos utilizing residual frames as the input data in 3D ConvNets. By replacing traditional stacked RGB frames with residual ones, 35.6% and 26.6% points improvements over top-1 accuracy can be achieved on the UCF101 and HMDB51 datasets when trained from scratch using ResNet-18-3D. We deeply analyze the effectiveness of this modality compared to normal RGB video clips, and find that better motion features can be extracted using residual frames with 3D ConvNets. Considering that residual frames contain little information of object appearance, we further use a 2D convolutional network to extract appearance features and combine them together to form a two-path solution. In this way, we can achieve better performance than some methods which even used an additional optical flow stream. Moreover, the proposed residual-input path can outperform RGB counterpart on unseen datasets when we apply trained models to video retrieval tasks. Huge improvements can also be obtained when the residual inputs are applied to video-based self-supervised learning methods, revealing better motion representation and generalization ability of our proposal.

Time Domain Channel Estimation and Equalization of CP-OTFS Under Multiple Fractional Dopplers and Residual Synchronization Errors

In the last few years, orthogonal time frequency space (OTFS) modulation has received significant attention as an alternative to OFDM especially for high mobilty scenarios. In this work, we develop a delay-Doppler domain embedded pilot based time domain channel estimation for cyclic prefix (CP)-OTFS in the presence of residual frame timing offset, carrier frequency offset and fractional multiple Doppler. One of the reasons for time domain processing is that the time domain channel representation is relatively more sparse as compared to its delay Doppler domain representation in the presence of residual synchronization errors. We also describe a time domain low complexity linear minimum mean square error (MMSE) equalization and successive interference cancellation (SIC) receiver for LDPC (low density parity check) coded CP-OTFS in this work. We further show the impact of residual frame timing offset, carrier frequency offset and fractional multiple Doppler on OTFS symbols. It is seen from the extensive Monte Carlo simulation results that the estimation and compensation methods presented here provide necessary resilience properties to OTFS. We bring out the tolerance of OTFS to such residual synchronization errors. It is further observed that the SIC is able to improve the performance of the system such that it almost matches that of the ideal knowledge based MMSE equalization. We also show the performance of RCP (reduced CP)-OTFS when used with the developed channel estimation and equalization algorithms. A unified signal processing flow for OTFS and orthogonal frequency division multiplexing (OFDM) is also described in this work to motivate studies on coexistence between the two as well as to encourage investigations on a seamless transition between OFDM and OTFS based systems for future adaptive air interface design.

Seismic performance of eccentrically braced frames designed to AISC341 and EC8 specifications

In the United States, steel eccentrically braced frames (EBFs) are designed according to AISC341 Specification while in Europe EC8 provisions are used. The European provisions were developed mostly based on the research works conducted in the US. While EBF design recommendations in AISC341 and EC8 have similarities, marked differences exist in the amount of lateral force reduction and distribution of member over-strength among the stories. In addition, the effect of link over-strength on non-dissipative members is treated differently in the two specifications. A study has been undertaken to evaluate the performance of EBFs designed according to AISC341 and EC8. Pursuant to this goal, EBF archetype buildings with 3, 6, and 9 stories were designed using US and European provisions taking into account different force reduction factors. The archetype buildings were subjected to 44 far-field ground motions at the maximum considered earthquake level as recommended by FEMA P695. Nonlinear time history analyses were conducted by OpenSees analysis platform to obtain the link rotation angle, residual frame drift and column axial forces. The results showed that using larger force reduction factors and disregarding the distribution of member over-strength among stories result in large residual frame drifts and non-uniform link rotation angles along the height of the building. Considering a force reduction factor of 5 and imposing a limit on link over-strength among the stories were found to improve the performance considerably. Design recommendations were developed to enhance the performance of EBFs by unifying the rules given in AISC341 and EC8. The study was complemented by investigating the behavior of EBFs subjected to near-field ground motions with pulse. The results showed that near-field ground motions with pulse produce much higher link rotation angles when compared with the demands produced by far-field ground motions.

Block Compressive Sensing Single-View Video Reconstruction Using Joint Decoding Framework for Low Power Real Time Applications

Several real-time visual monitoring applications such as surveillance, mental state monitoring, driver drowsiness and patient care, require equipping high-quality cameras with wireless sensors to form visual sensors and this creates an enormous amount of data that has to be managed and transmitted at the sensor node. Moreover, as the sensor nodes are battery-operated, power utilization is one of the key concerns that must be considered. One solution to this issue is to reduce the amount of data that has to be transmitted using specific compression techniques. The conventional compression standards are based on complex encoders (which require high processing power) and simple decoders and thus are not pertinent for battery-operated applications, i.e., VSN (primitive hardware). In contrast, compressive sensing (CS) a distributive source coding mechanism, has transformed the standard coding mechanism and is based on the idea of a simple encoder (i.e., transmitting fewer data-low processing requirements) and a complex decoder and is considered a better option for VSN applications. In this paper, a CS-based joint decoding (JD) framework using frame prediction (using keyframes) and residual reconstruction for single-view video is proposed. The idea is to exploit the redundancies present in the key and non-key frames to produce side information to refine the non-key frames’ quality. The proposed method consists of two main steps: frame prediction and residual reconstruction. The final reconstruction is performed by adding a residual frame with the predicted frame. The proposed scheme was validated on various arrangements. The association among correlated frames and compression performance is also analyzed. Various arrangements of the frames have been studied to select the one that produces better results. The comprehensive experimental analysis proves that the proposed JD method performs notably better than the independent block compressive sensing scheme at different subrates for various video sequences with low, moderate and high motion contents. Also, the proposed scheme outperforms the conventional CS video reconstruction schemes at lower subrates. Further, the proposed scheme was quantized and compared with conventional video codecs (DISCOVER, H-263, H264) at various bitrates to evaluate its efficiency (rate-distortion, encoding, decoding).

No-Reference Video Quality Assessment Based on Visual Memory Modeling

Objective video quality assessment plays an important role in a variety of video processing applications such as video compression, transmission, visualization, and display. This paper proposes a no-reference video quality assessment model based on visual memory understanding. Inspired by the findings of neuroscience researchers, who argue there is a large overlap between the active human brain area when performing video quality assessment and saliency detection tasks, saliency maps are employed here to assist the quality assessment. To this end, we first generate the saliency maps using CLBP (Complete Local Binary Patterns) features of the residual frames. Then, a model of visual memory is created from the statistics of saliency maps. This is followed by learning the video quality from the visual memory, saliency, and frame features through a support vector regression pipeline. The experimental results on the state-of-the-art LIVE and SJTU video datasets indicate that the proposed no-reference video quality assessment algorithm is effective and performs statistically better than several other state-of-the-art approaches.