Sequence Of Frames Research Articles

Adaptive Buffer Capture and Storage for High-Speed Cameras

The output speed of a high-resolution camera with a fast frame rate is high, reaching 800 MB/s or even more. Consequently, the abundant data recorded can easily cause data overflow during transmission and frame loss during storage. To address this problem of data overflow and frame loss while achieving reliable acquisition rates, herein, we design a high-speed video adaptive transmission and storage method based on the camera link transmission protocol. The system uses the camera link transmission protocol and high-speed cameras for data acquisition. During data transmission, memory space is employed as the data buffer pool, an adaptive memory buffer pool is designed using the hardware and acquisition parameters, and a circular read/write buffer, combined with a block device read/write method, is used inside the buffer pool for data transmission. For data storage, input/output completion port asynchronous storage mixed with frame sequence number filtering is employed to write data, and the adaptive buffer of the hard disk is used to achieve efficient data writing. The experimental results reveal that the proposed method can realize efficient and stable writing of data on conventional industrial cameras based on the camera link interface, effectively addressing the problem of frame loss during data storage. Moreover, the stored data can be transferred efficiently with less resource occupation, thereby meeting the application requirements of high-precision vision measurement systems and demonstrating broad application value.

Image Enhancement and Brightness Equalization Algorithms in Low Illumination Environment Based on Multiple Frame Sequences

Image Enhancement and Brightness Equalization Algorithms in Low Illumination Environment Based on Multiple Frame Sequences

Retrieving Object Motions From Coded Shutter Snapshot in Dark Environment.

Video object detection is a widely studied topic and has made significant progress in the past decades. However, the feature extraction and calculations in existing video object detectors demand decent imaging quality and avoidance of severe motion blur. Under extremely dark scenarios, due to limited sensor sensitivity, we have to trade off signal-to-noise ratio for motion blur compensation or vice versa, and thus suffer from performance deterioration. To address this issue, we propose to temporally multiplex a frame sequence into one snapshot and extract the cues characterizing object motion for trajectory retrieval. For effective encoding, we build a prototype for encoded capture by mounting a highly compatible programmable shutter. Correspondingly, in terms of decoding, we design an end-to-end deep network called detection from coded snapshot (DECENT) to retrieve sequential bounding boxes from the coded blurry measurements of dynamic scenes. For effective network learning, we generate quasi-real data by incorporating physically-driven noise into the temporally coded imaging model, which circumvents the unavailability of training data and with high generalization ability on real dark videos. The approach offers multiple advantages, including low bandwidth, low cost, compact setup, and high accuracy. The effectiveness of the proposed approach is experimentally validated under low illumination vision and provide a feasible way for night surveillance.

Highly Similar Average Collateral Effect of Synonymous Mutations Across Alternative Reading Frames: A Potential Role In Evolvability

Abstract Synonymous mutations in a protein coding gene lead to a remarkably similar average “collateral” mutation effect size across alternative reading frames (1). Here we quantify the rarity of this feature among possible block structure codes as 0.77%. Then we develop a simple model of evolutionary search with two types of mutation. Across different mutation step sizes and ratios of the two types, the fitness-maximizing region corresponds to using a single average mutation value. The analogous constant average collateral mutation effect observed for the standard genetic code may likewise facilitate evolutionary search in alternative frame sequences.

A hybrid neural network-based intelligent body posture estimation system in sports scenes.

Body posture estimation has been a hot branch in the field of computer vision. This work focuses on one of its typical applications: recognition of various body postures in sports scenes. Existing technical methods were mostly established on the basis of convolution neural network (CNN) structures, due to their strong visual information sensing ability. However, sports scenes are highly dynamic, and many valuable contextual features can be extracted from multimedia frame sequences. To handle the current challenge, this paper proposes a hybrid neural network-based intelligent body posture estimation system for sports scenes. Specifically, a CNN unit and a long short-term memory (LSTM) unit are employed as the backbone network in order to extract key-point information and temporal information from video frames, respectively. Then, a semi-supervised learning-based computing framework is developed to output estimation results. It can make training procedures using limited labeled samples. Finally, through extensive experiments, it is proved that the proposed body posture estimation method in this paper can achieve proper estimation effect in real-world frame samples of sports scenes.

Video Frame Prediction by Joint Optimization of Direct Frame Synthesis and Optical-Flow Estimation

Video prediction is the problem of generating future frames by exploiting the spatiotemporal correlation from the past frame sequence. It is one of the crucial issues in computer vision and has many real-world applications, mainly focused on predicting future scenarios to avoid undesirable outcomes. However, modeling future image content and object is challenging due to the dynamic evolution and complexity of the scene, such as occlusions, camera movements, delay and illumination. Direct frame synthesis or optical-flow estimation are common approaches used by researchers. However, researchers mainly focused on video prediction using one of the approaches. Both methods have limitations, such as direct frame synthesis, usually face blurry prediction due to complex pixel distributions in the scene, and optical-flow estimation, usually produce artifacts due to large object displacements or obstructions in the clip. In this paper, we constructed a deep neural network Frame Prediction Network (FPNet-OF) with multiple-branch inputs (optical flow and original frame) to predict the future video frame by adaptively fusing the future object-motion with the future frame generator. The key idea is to jointly optimize direct RGB frame synthesis and dense optical flow estimation to generate a superior video prediction network. Using various real-world datasets, we experimentally verify that our proposed framework can produce high-level video frame compared to other state-of-the-art framework.

Deep learning MRI analysis for automated knee injury diagnosis

Magnetic Resonance Imaging (MRI) plays a pivotal role in non-invasively diagnosing knee injuries. This research focuses on cost-effective, efficient solutions for enhancing automated knee injury detection in MRI scans. The study aims to boost diagnostic accuracy for abnormalities, Anterior Cruciate Ligament (ACL) tears, and meniscal tears using advanced deep learning techniques. Transfer learning is employed, combining pretrained neural networks with transfer models. AlexNet and SqueezeNet are explored as feature extraction architectures, assessing the attention mechanism and maxpooling for sequence reduction. This yields four models, MRNet, MRNet-Squeeze, MRNet-Attend, and MRNet- SqueezeAttend. The primary evaluation metric is the Area Under the ROC Curve (AUC), providing a comprehensive assessment by averaging AUC scores for abnormal, ACL tear, and meniscus tear labels. The initial MRNet achieves the highest AUC (0.940) for anomaly detection, while MRNet-Squeeze excels in diagnosing ACL damage. MRNet- SqueezeAttend achieves the highest AUC (0.885) for meniscus tears. The ensemble of all four models outperforms individual models with an outstanding average AUC of 0.931. Each model exhibits unique strengths. MRNet excels at spotting anomalies, MRNet-Squeeze accurately detects ACL tears, and MRNet-SqueezeAttend excels in identifying meniscal tears. Notably, the ensemble leverages these diverse strengths to deliver cutting-edge results for all injury types. Further investigation reveals varying correlations between model-specific predictions and different diagnosis or sequence combinations. Scrutinizing the MRI sequence frames capturing the models' attention identifies key contributors to the diagnosis.

Dual-Modality Space-Time Memory Network for RGBT Tracking

RGBT tracking is rapidly developing due to its complementary advantages of RGB and thermal frames. Existing methods with high accuracy track at a lower speed, and do not make full use of the hierarchical information in the feature extraction and the historical information of the sequences. To address these issues, a novel dual-modality space-time memory (DMSTM) network is proposed for robust RGBT tracking. Specifically, DMSTM is divided into three modules. The first module is the dual-modality backbone that utilizes both shallow and deep information by aggregating feature maps of dimensional changes during downsampling. Another module is the space-time memory reader with bimodal fusion. It aggregates features of historical and current frames to share information in the time domain. The last module is the siamese head network, which computes the predicted loss sum of the two modalities and back-propagates it. This avoids degrading the tracking performance due to sequence frame pairs where the training targets are not perfectly aligned. Extensive experiments on three RGBT benchmark datasets show that the performance and efficiency of the proposed DMSTM exceed that of state-of-the-art methods while running at 27.6 FPS.

Online Video Super-Resolution With Convolutional Kernel Bypass Grafts

Deep learning-based models have achieved remarkable performance in video super-resolution (VSR) in recent years, but most of these models are less applicable to online video applications. These methods solely consider the distortion quality and ignore crucial requirements for online applications, e.g., low latency and low model complexity. In this paper, we focus on online video transmission in which VSR algorithms are required to generate high-resolution video sequences frame by frame in real time. To address such challenges, we propose an extremely low-latency VSR algorithm based on a novel kernel knowledge transfer method, named the convolutional kernel bypass graft (CKBG). First, we design a lightweight network structure that does not require future frames as inputs and saves extra time for caching these frames. Then, our proposed CKBG method enhances this lightweight base model by bypassing the original network with “kernel grafts”, which are extra convolutional kernels containing the prior knowledge of the external pretrained image SR models. During the testing phase, we further accelerate the grafted multibranch network by converting it into a simple single-path structure. The experimental results show that our proposed method can process online video sequences up to 110 FPS with very low model complexity and competitive SR performance.

Automatic Face Mask Detection on Gates to Combat Spread Of Covid_19

The COVID-19 pandemic has spread across the globe, hitting almost every country. To stop the spread of the COVID-19 pandemic, this article introduces face mask detection on a gate to assure the safety of Instructors and students in both class and public places. This work aims to distinguish between faces with masks and without masks. A deep learning algorithm You Only Look Once (YOLO) V5 is used for face mask detection and classification. This algorithm detects the faces with and without masks using the video frames from the surveillance camera. The model trained on over 800 video frames. The sequence of a video frame for face mask detection is fed to the model for feature acquisition. Then the model classifies the frames as faces with a mask and without a mask. We used loss functions like Generalize Intersection of Union for abjectness and classification accuracy. The datasets used to train the model are divided as 80% and 20% for training and testing, respectively. The model has provided a promising result. The result found shows accuracy and precision of 95% and 96%, respectively. Results show that the model performance is a good classifier. The successful findings indicate the suggested work's soundness.
 

Motrcycling-Net: A Segmentation Approach for Detecting Motorcycling near Misses

Commercial motorcycling is one of the economical means of transportation in many countries, although many perceive it as a dangerous means of transportation, which is affirmed by the number of casualties recorded daily. This life threatening record has greatly hindered continuous support for commercial motorcycling as an affordable means of transportation. Information retrieved from near miss datasets can be a telltale of potential hazards and their prevention. However, many researchers have come up with different definitions for near misses, and this has created a gap in applying the right method to near misses, thereby making it statistically difficult to address the situation for a safer commercial motorcycling. In this paper, we present near misses as corrective and preventive measures to safety events. Our focus is on the risk factors of commercial motorcycling near miss incidents, which we address by proposing a near miss detection framework based on deep learning and its models. Video streams of near miss datasets containing motorcycling in different scenes were collected for the experiment. We employed YOLOv4-DeepSort model for the detection and tracking tasks, and the tracked images and identity information were stored. Every 1s, the sequence of image was fetched into the VGG16-BiLSTM model (VGG16 and BiLSTM were used for extraction of image feature information and near misses recognition respectively). We evaluate the method by testing 444 sequential video frames of motorcycling near miss incidents in urban environment, and approximately 96% recognition accuracy rate is achieved. The results of the study indicate practicality for automatic detection of motorcycling near misses in urban environment, and it could assist in providing resourceful technical reference for analyzing the risk factors of motorcycling near misses.

Multi-channel network: Constructing efficient GCN baselines for skeleton-based action recognition

Skeleton-based action sequences are widely used for human behaviour understanding due to their compact characteristics. Most existing work designed Graph Convolutional Networks and integrated multiple input channels rather than the original motion sequence to improve the final performance. However, few of them have been reported on the detailed effects of such multiple input channels. In contrast to them, we systemically study the impact of different input channels and construct a more efficient GCN framework. We have identified the complementary effect between the local frame channel and global sequence channel, which is essential to improve the action recognition accuracy. By coupling local frame and global sequence information with a classical spatial–temporal graph neural network, e.g. MS-G3D, it achieves competitive performance compared with SOTA methods on challenging benchmark datasets. Related code would be available at https://github.com/movearbitrarily/multi-stream.

CAFI-AR

Freehand interaction enhances user experience, allowing one to use bare hands to manipulate virtual objects in AR. Yet, it remains challenging to accurately and efficiently detect contacts between real hand and virtual object, due to the imprecise captured/estimated hand geometry. This paper presents CAFI-AR, a new approach for Contact-Aware Freehand Interaction with virtual AR objects, enabling us to automatically detect hand-object contacts in real-time with low latency. Specifically, we formulate a compact deep architecture to efficiently learn to predict hand action and contact moment from sequences of captured RGB images relative to the 3D virtual object. To train the architecture for detecting contacts on AR objects, we build a new dataset with 4,008 frame sequences, each with annotated hand-object interaction information. Further, we integrate CAFI-AR into our prototyping AR system and develop various interactive scenarios, demonstrating fine-grained contact-aware interactions on a rich variety of virtual AR objects, which cannot be achieved by existing AR interaction approaches. Lastly, we also evaluate CAFI-AR, quantitatively and qualitatively, through two user studies to demonstrate its effectiveness in terms of accurately detecting the hand-object contacts and promoting fluid freehand interactions

VStyclone: Real-time Chinese voice style clone

This paper proposes a novel Chinese speech cloning model named VStyclone, which consists of three stages: multi-speaker training, target speaker encoding, and target speaker synthesis. In this work, we design an efficient tone extractor, which can reallocate resources to the sequences of log-mel spectrogram frames obtained from multiple speakers’ speech, thus allowing the network to learn multiple speakers’ features differently. This approach allows the network to focus more on the voice features of the target speaker and extract the target features accurately. To cluster the voices of the same speaker and sparse the voices of different speakers, we build an optimal softmax loss to optimize the model. Then, we develop a style synthesizer, which adopts the idea of transformer instead of recurrent neural network, so that the model can not only process text information in parallel, but also improve the model's ability to process long-distance contextual information. Meanwhile, we embed a style extraction module in the style synthesizer to dynamically capture style ranges in an unsupervised manner. In addition, the VStyclone model uses generative adversarial networks as the base generation model of the vocoder to improve the generation speed, which runs 1.2 times faster than the real-time generation speed on CPU, and finally the VStyclone model obtains the SOTA effect.

AN APPROACH TO SUBPIXEL ACCURACY WIDENING CRACK WIDTH DETERMINATION IN IMAGE SEQUENCES

Abstract. As an extension to existing work on crack detection and subpixel accuracy crack width determination as a tool for civil engineering material testing, the paper shows an algorithmic approach to handle widening cracks with relative rotations between related crack borders. In the first time step under zero-load, a set of points to be tracked through consecutive frames of an image sequence is defined. Then, subpixel-precise displacement fields are computed for the image data of the following time steps using an 8-parameter least-squares matching approach. The points are triangulated into a mesh, and the changes of the inner geometry of the triangles are considered with a mathematical model assuming a split of each triangle. With this model, subpixel-precise deformation vectors are derived. Crack candidates are determined by a thresholding applied to the vectors’ lengths. After an estimation of the crack normal, a decomposition of the deformation vectors is applied, allowing to compute crack widths and shear movements. As a novel contribution to the technique, a model extension is proposed for the case of a relative rotation between the crack borders in order to reduce systematic errors. The model includes two separate rigid transformations for each crack side.

Identification of a fatty acid synthase gene (FAS1) from Laodelphax striatellus planthoppers contributing to fecundity.

Fatty acid synthase (FAS) is a multifunctional enzyme that plays an important role in the formation of fatty acids. The fatty acids take part in many processes, such as cell signaling and energy metabolism, and in insects they are important in both cuticular hydrocarbon (CHC) formation and reproduction. Here we characterized the sequence structure and function of an FAS from the small brown planthopper (SBPH), Laodelphax striatellus. The full-length open reading frame (ORF) sequence of LsFAS1 was 7122bp, encoding a predicted protein of 2373 amino acid residues. There were 7 functional domains in the LsFAS1 protein sequence. Gene expression screening by real-time quantitative polymerase chain reaction (RT-qPCR) showed that LsFAS1 was expressed in all developmental stages. Relative expression was highest at the 4th-instar and female adult stages. Among different tissues, the expression level of LsFAS1 in the ovary was the highest. Phylogenetic analysis showed that LsFAS1 clustered in a clade with 2 FASs from Nilaparvata lugens. Furthermore, these 3 FASs are related to cockroach BgFAS and locust LmFAS. After RNA interference-mediated knock-down, most treated insects died at eclosion. In addition, the lifespan of dsFAS1-treated female adults was shorter than that of the dsGFP-injected control, and offspring production decreased. Also, the expression of vitellogenin (Vg) and vitellogenin receptor (VgR) genes decreased. Virgin females dissected at days2 and 4 post-eclosion showed many matured oocytes in planthoppers treated with dsGFP but not with dsFAS1. These data highlight the importance of LsFAS1 in SBPH, including a role in reproduction.

A Note on the Invertibility of the Gabor Frame Operator on Certain Modulation Spaces

We consider Gabor frames generated by a general lattice and a window function that belongs to one of the following spaces: the Sobolev space V_1 = H^1(mathbb {R}^d), the weighted L^2-space V_2 = L_{1 + |x|}^2(mathbb {R}^d), and the space V_3 = mathbb {H}^1(mathbb {R}^d) = V_1 cap V_2 consisting of all functions with finite uncertainty product; all these spaces can be described as modulation spaces with respect to suitable weighted L^2 spaces. In all cases, we prove that the space of Bessel vectors in V_j is mapped bijectively onto itself by the Gabor frame operator. As a consequence, if the window function belongs to one of the three spaces, then the canonical dual window also belongs to the same space. In fact, the result not only applies to frames, but also to frame sequences.

Multi-scale local-temporal similarity fusion for continuous sign language recognition

Continuous sign language recognition (cSLR) is a public significant task that transcribes a sign language video into an ordered gloss sequence. It is important to capture the fine-grained gloss-level details, since there is no explicit alignment between sign video frames and the corresponding glosses. Among the past works, one promising way is to adopt a one-dimensional convolutional network (1D-CNN) to temporally fuse the sequential frames. However, CNNs are agnostic to similarity or dissimilarity, and thus are unable to capture local consistent semantics within temporally neighboring frames. To address the issue, we propose to adaptively fuse local features via temporal similarity for this task. Specifically, we devise a Multi-scale Local-Temporal Similarity Fusion Network (mLTSF-Net) as follows: 1) In terms of a specific video frame, we firstly select its similar neighbours with multi-scale receptive regions to accommodate different lengths of glosses. 2) To ensure temporal consistency, we then use position-aware convolution to temporally convolve each scale of selected frames. 3) To obtain a local-temporally enhanced frame-wise representation, we finally fuse the results of different scales using a content-dependent aggregator. We train our model in an end-to-end fashion, and the experimental results on RWTH-PHOENIX-Weather 2014 datasets (RWTH) demonstrate that our model achieves competitive performance compared with several state-of-the-art models.

An adaptive frame selection network with enhanced dilated convolution for video smoke recognition

Recognizing smoke from surveillance videos is crucial to achieve robust fire detection due to the rich temporal cues in video data. However, the slow-moving smoke makes it difficult to learn complete motion information of smoke. In addition, the widely-used max/average pooling operations in smoke recognition models may lead to the loss of spatial detail information. To solve the above issues, we propose an adaptive frame selection network (AFSNet) with enhanced dilated convolution for video smoke recognition. Specifically, first, to reduce information redundancy for learning discriminative feature representations, we propose an adaptive frame selection convolution to automatically select the most useful frames in the image sequence. Second, to utilize all the pixels for reducing the loss of detail information and carrying over global information, we propose an enhanced dilated convolution by considering the pixels with high responses as well as the local region to learn the contribution of each pixel to the output automatically. Finally, a feature extraction module without any average and max pooling operations is designed to learn multi-scale, context, and spatiotemporal information simultaneously. Experiments show that our model achieves the highest detection rate of 0.9673 and the lowest false alarm rate of 0.0316 on the SRSet dataset, and achieves the highest F1-score accuracy of 0.85, 0.86, and 0.91 on the three subsets of the RISE dataset, respectively.

VEHICLE TRACKING AND SPEED ESTIMATION UNDER MIXED TRAFFIC CONDITIONS USING YOLOV4 AND SORT: A CASE STUDY OF HANOI

This paper presents a method to estimate vehicle speed automatically, including cars and motorcycles under mixed traffic conditions from video sequences acquired with stationary cameras in Hanoi City of Vietnam. The motion of the vehicle is detected and tracked along the frames of the video sequences using YOLOv4 and SORT algorithms with a custom dataset. In the method, the distance traveled by the vehicle is the length of virtual point-detectors, and the travel time of the vehicle is calculated using the movement of the centroid over the entrance and exit of virtual point-detectors (i.e., region of interest), and then the speed is also estimated based on the traveled distance and the travel time. The results of two experimental studies showed that the proposed method had small values of MAPE (within 3%), proving that the proposed method is reliable and accurate for application in real-world mixed traffic environments like Hanoi, Vietnam.