Key Frames Research Articles

Spatial-Temporal Co-Attention Learning for Diagnosis of Mental Disorders From Resting-State fMRI Data.

Neuroimaging techniques have been widely adopted to detect the neurological brain structures and functions of the nervous system. As an effective noninvasive neuroimaging technique, functional magnetic resonance imaging (fMRI) has been extensively used in computer-aided diagnosis (CAD) of mental disorders, e.g., autism spectrum disorder (ASD) and attention deficit/hyperactivity disorder (ADHD). In this study, we propose a spatial-temporal co-attention learning (STCAL) model for diagnosing ASD and ADHD from fMRI data. In particular, a guided co-attention (GCA) module is developed to model the intermodal interactions of spatial and temporal signal patterns. A novel sliding cluster attention module is designed to address global feature dependency of self-attention mechanism in fMRI time series. Comprehensive experimental results demonstrate that our STCAL model can achieve competitive accuracies of 73.0 ± 4.5%, 72.0 ± 3.8%, and 72.5 ± 4.2% on the ABIDE I, ABIDE II, and ADHD-200 datasets, respectively. Moreover, the potential for feature pruning based on the co-attention scores is validated by the simulation experiment. The clinical interpretation analysis of STCAL can allow medical professionals to concentrate on the discriminative regions of interest and key time frames from fMRI data.

Motion In-Betweening via Deep ∆-Interpolator.

We show that the task of synthesizing human motion conditioned on a set of key frames can be solved more accurately and effectively if a deep learning based interpolator operates in the delta mode using the spherical linear interpolator as a baseline. We empirically demonstrate the strength of our approach on publicly available datasets achieving state-of-the-art performance. We further generalize these results by showing that the ∆-regime is viable with respect to the reference of the last known frame (also known as the zero-velocity model). This supports the more general conclusion that operating in the reference frame local to input frames is more accurate and robust than in the global (world) reference frame advocated in previous work.

User Engagement Detection-Based Financial Technology Advertising Video Effectiveness Evaluation

With the rapid advancement of financial technology, an increasing number of related advertisements have received widespread attention. User engagement detection during the advertisement viewing process directly reflects the effectiveness of the advertising video. Therefore, detecting user engagement during the advertisement viewing process has become a crucial issue. However, traditional engagement detection methods often require significant computational resources, significantly reducing their practicality. To address this issue, the authors propose a method to effectively detect user engagement by fully integrating multiple relatively practical models. Specifically, the authors extract key frame images from user face video and perform super-resolution reconstruction of them. Then image pyramid matching is used to achieve user engagement detection. Finally, the authors establish a reasonable database and conduct sufficient experiments based on it. Experimental results demonstrate that this proposed method has realistic engagement detection accuracy, and the design of multiple steps is also valid.

Air-ground multi-agent system cooperative navigation based on factor graph optimization SLAM

In this study, a cooperative navigation algorithm centered on factor graph optimization—simultaneous localization and mapping (FGO-SLAM) is presented for an air-ground multi-agent system. The algorithm prioritizes the control of error statuses during the position and attitude estimation procedure throughout the entire back-end optimization process. In the conventional extended kalman filtering (EKF) algorithm, periodic cumulative errors may arise, introducing uncertainty to the estimation process. The application of the FGO algorithm not only mitigates deviation but also stabilizes errors, thereby eliminating the accumulation of periodic errors. In comparison to the practical EKF-SLAM, FGO-SLAM serves as a semi-offline optimization system that leverages key frames to minimize computational load. During multi-agent simulations, when two or more agents have overlapping field views, landmark data is merged, enhancing the optimization effectiveness. Through simulation experiments, the proposed algorithm demonstrates a 40% reduction in position error and a 41% reduction in attitude error, affirming the efficacy of FGO-SLAM for cooperative navigation.

Multi-View Jujube Tree Trunks Stereo Reconstruction Based on UAV Remote Sensing Imaging Acquisition System

High-quality agricultural multi-view stereo reconstruction technology is the key to precision and informatization in agriculture. Multi-view stereo reconstruction methods are an important part of 3D vision technology. In the multi-view stereo 3D reconstruction method based on deep learning, the effect of feature extraction directly affects the accuracy of reconstruction. Aiming at the actual problems in orchard fruit tree reconstruction, this paper designs an improved multi-view stereo structure based on the combination of remote sensing and artificial intelligence to realize the accurate reconstruction of jujube tree trunks. Firstly, an automatic key frame extraction method is proposed for the DSST target tracking algorithm to quickly recognize and extract high-quality data. Secondly, a composite U-Net feature extraction network is designed to enhance the reconstruction accuracy, while the DRE-Net feature extraction enhancement network improved by the parallel self-attention mechanism enhances the reconstruction completeness. Comparison tests show different levels of improvement on the Technical University of Denmark (DTU) dataset compared to other deep learning-based methods. Ablation test on the self-constructed dataset, the MVSNet + Co U-Net + DRE-Net_SA method proposed in this paper improves 20.4% in Accuracy, 12.8% in Completion, and 16.8% in Overall compared to the base model, which verifies the real effectiveness of the scheme.

Multi-stream network with key frame sampling for human action recognition

Multi-stream network with key frame sampling for human action recognition

UnifiedTT: Visual tracking with unified transformer

Target tracking is an important research task in computer vision. Existing tracking algorithms based on Siamese networks often suffer from the problem of information redundancy between adjacent frames and a lack of ability to capture global dependencies. When similar backgrounds appear around the target, the tracking performance usually significantly decreases. Although target tracking algorithms based on deep convolution and the Transformer have partially addressed these issues, achieving a good balance between the two remains a challenge. In this work, we propose a unified convolution and self-attention Siamese network for target tracking. By utilizing a feature extraction backbone network based on integrated convolution and self-attention styles, we are able to capture globally important regions and key frames while greatly reducing local redundant computations, thereby improving tracking performance. We apply this approach to the task of target tracking and enhance the feature extraction capabilities of both the target template and search template. Experimental results show that our proposed tracking algorithm outperforms some recent classical tracking algorithms, especially achieving improvements of 10.7 % on the high-diversity dataset GOT-10K and 24.7 % on the large-scale and high-quality LaSOT dataset.

Retracted: Key Frame Extraction Method for Minors’ Participation in Online Short-Form Video from the Perspective of Government Administration

Retracted: Key Frame Extraction Method for Minors’ Participation in Online Short-Form Video from the Perspective of Government Administration

Multi-temporal dependency handling in video smoke recognition: A holistic approach spanning spatial, short-term, and long-term perspectives

Accurately recognizing video-based smoke is still a profoundly challenging task due to the special characteristics of smoke, such as non-rigid morphology, semi-transparent representations, serious interferences (namely non-smoky white steams and clouds etc.), and so on. In this study, we present the Spatio-Temporal Local-Enhanced Network (STENet), which is meticulously designed to handle the spatio-temporal dependencies of smoke video in multi-temporal views, effectively synthesizing spatial, long-term, and short-term smoke features. To account for the semi-translucent nature of smoke, we introduce a short-term temporal branch within STENet that models the transient temporal dependencies between adjacent frames, thereby distinguishing smoke from visually similar interferences such as clouds and water steam. Furthermore, the representation of spatial and short-term temporal features, which capture the instantaneous spatio-temporal characteristics of a single smoke plume, can be fine-grained aligned and mutually guided, which is accomplished through our Collaborative Alignment(CoAM). Moreover, to address the unique drifting characteristics and dynamic variations of non-rigid objects across different temporal scales, we propose a new Prioritized Local-enhanced Transformer (PLT) that selectively focuses on key frame subsets for capturing long-term temporal dependencies. Experiments conducted on benchmarks demonstrate that the proposed STENet achieves state-of-the-art results on large-scale video dataset for recognizing industrial smoke emissions (RISE) and forest fire surveillance video (FSmoke). In particular, our STENet achieves 0.893 F-score on RISE dataset with 6.9M parameters and 3.86G FLOPs, being 4×-18× smaller and using 5×-51× fewer FLOPs than previous baseline models.

Investigating Methods of Searching for Key Frames in Video Flow with the Use of Neural Networks for Search Systems

Various methods of video content data analysis are presented, compared, and evaluated in this paper. Due to the analysis, the most effective strategies for video data processing involve searching for key frames within the video stream. The examined methods are categorized into consistent comparison, global comparison based on clustering, and event/object-based methodologies. Key techniques such as sequence search, classification, frame decoding, and anomaly detection are singled out as particularly valuable for comparison and matching tasks. The research further reveals that artificial intelligence and machine learning-driven methods reign supreme in this domain, with deep learning approaches outperforming traditional techniques. The employment of convolutional neural networks and attention mechanisms to capture the temporal intricacies across variable scopes is especially noteworthy. Additionally, leveraging the Actor-Critic model within a Generative Adversarial Network framework has shown encouraging outcomes. A significant highlight of the study is the proposed approach which incorporates modified Independent Recurrent Neural Networks (IndRNN) complemented by an attention mechanism. The enhancement using mathematical tools, notably the standard deviation, for key frame detection, exemplifies the potential of integrating analytical instruments to refine the system's precision. Such advancements, as presented in this research, pave the way for substantial enhancements in information systems tailored for video content analysis and source identification.

Homeostasis and information processing: The key frames for the thermodynamics of biological systems

Life is a natural phenomenon ineluctably subject to the laws and principles of physics. In this framework, thermodynamics has a crucial role, since living beings are structured on a molecular and cellular basis that can only be maintained with extensive energy consumption. This imposes that living beings are necessarily open systems. But the survival of each type of organism depends on the relative stability of certain essential variables, even in the presence of the disturbances to which they are subjected. The stability of these variables is relative in the sense that they have a narrow range of variation. This stability of the essential variables is a consequence of refined control mechanisms developed in the course of evolution, that lead to the condition called homeostasis. This homeostasis requires that control mechanisms process the various types of information related to the internal structure of the organism and its environment. Consequently, a biological system, through information processing aimed at guiding the mechanisms that maintain its homeostasis, manages the conditions imposed by the principles of thermodynamics, obtaining the most efficient use of energy possible and keeping entropic degradation controlled. In this article, we discuss the close links between thermodynamics, homeostasis and the information processing necessary to maintain homeostasis.

Empower smart cities with sampling-wise dynamic facial expression recognition via frame-sequence contrastive learning

In the construction of smart cities, facial expression analysis plays a crucial role. It can be used in traffic monitoring systems to alleviate traffic pressure by analyzing the emotional states of drivers and passengers. In the field of smart healthcare, it can provide more precise treatment and services to patients. In the realm of social entertainment, it can offer more intelligent and personalized interactions. In summary, the application of emotion computing technology will play an increasingly significant role in the development of smart cities in the future. In the task of dynamic facial expression recognition (DFER), analyzing the spatial–temporal features of video sequences has become a common research approach. However, facial expression sequences often contain a significant number of neutral frames and noisy frames, potentially increasing computational costs and reducing performance. Effectively extracting key frames for spatial–temporal feature analysis is a critical aspect of dynamic facial expression recognition. To address this issue, we proposed a sampling-wise dynamic facial expression recognition via frame-Sequence contrastive learning method, called SW-FSCL. The SW-FSCL method aims to improve the performance of DFER by using intelligent dual-stream sampling strategies and frame-sequence contrastive learning, extract key frame and reduce the impact of neutral frames and noisy frames. We proposed a key frame proposal (KFP) block to analyze the spatial–temporal features of sequences, calculating weight ratios for key frame extraction. Due to potential information loss in long sequences, we introduce a temporal aggregation (TA) block to prevent data loss and ensure the integrity of temporal information. The experimental results provide compelling evidence that the proposed approach not only outperforms all current state-of-the-art algorithms on two widely-used benchmark datasets (DFEW, FERV39k), but also visualization results produces insights into the interpretability of the SW-FSCL method.

Feature Matching-Based Undersea Panoramic Image Stitching in VR Animation

The continuous development of virtual reality animation has brought people a new viewing experience. However, there is still a large research space for the construction of virtual scenes. Underwater scenes are complex and diverse, and to obtain more realistic virtual scenes, it is necessary to use video panoramic images as reference modeling in advance. To this end, the study uses the [Formula: see text]-means clustering method to extract key frames from underwater video, and adaptively adjusts the number of clusters to improve the extraction algorithm according to the differences in features. To address the problems of low contrast and severe blurring in underwater images, the study uses an improved non-local a priori recovery method to achieve the recovery process of underwater images. Finally, the final underwater panoramic image is obtained by fading-out image fusion and frame to stitching image synthesis strategy. The experimental analysis shows that the runtime of Model 1 is 21.46[Formula: see text]s, the root mean square error value is 1.89, the structural similarity value is 0.9678, and the average gradient value is 12.59. It can achieve efficient and high-quality panoramic image generation.

Video Semantic Segmentation Network with Low Latency Based on Deep Learning

Recently, new advances in deep learning algorithms have yielded some fascinating results in the field of computer vision technology. As a result, it can now perform activities that formerly required the use of human vision and the brain. Classification, object identification, and semantic segmentation have all seen substantial advancements in deep learning architecture in the last few years. For still images and movies, there has been a major advancement in the field of semantic segmentation. In practical uses like autonomous vehicles, segmenting semantic video continues to be difficult due to high-performance standards, the high cost of convolutional neural networks (CNNs), and the significant need for low latency. An effective machine-learning environment will be developed to meet the performance and latency challenges outlined above. The use of deep learning architectures like SegNet and FlowNet2.0 on the CamVid dataset enables this environment to conduct pixel-wise semantic segmentation of video properties while maintaining low latency. As a result, it is ideally suited for real-world applications since it takes advantage of both SegNet and FlowNet topologies. The decision network determines whether an image frame should be processed by a segmentation network or an optical flow network based on the expected confidence score. In conjunction with adaptive scheduling of the key frame approach, this technique for decision-making can help to speed up the procedure. Using the ResNet50 SegNet model, a mean Intersection on Union (IoU) of "54.27 percent" and an average frame per second of "19.57" were observed. Aside from decision network and adaptive key frame sequencing, it was discovered that FlowNet2.0 increased the frames processed per second9(fps) to "30.19" on GPU with a mean IoU of "47.65%". Because the GPU was utilized "47.65%" of the time, this resulted. There has been an increase in the speed of the Video semantic segmentation network without sacrificing quality, as demonstrated by this improvement in performance.

Research on the Digital Preservation of Intangible Cultural Heritage of Folk Dance Art Category

Abstract In this paper, based on the rapid head generation technique of the model library, we obtain a frontal 2D face photo, mark 13 feature points selected from 7 parts, such as both eyes, nose, mouth, etc., to determine the approximate position of the whole facial features, simulate the real color of the face according to the texture mapping of the face, and at the same time, we add the texture features of the target face in the to the matching model, to achieve the mapping of overlaying texture to the neutral texture and the fusion of skin color. Using HigherHRNet network extraction to obtain the coordinate information of the joint points of the intangible cultural heritage dance inheritors and their heat maps, the obtained key frames of the dance features are connected in a certain order to obtain the synthesized folk dance video with gesture estimation. Combining semantic segmentation of keyframes and style rendering, the visual image of the dance is designed, and the intangible cultural heritage of folk dance is analyzed through examples. The results show that this paper’s method achieves more than 90% recognition accuracy on all three datasets and more than 94.8% recognition accuracy on the folk dance movement dataset. On the evaluation of folk dance movements, the average distance calculated by this paper’s algorithm is the largest, 95.7, and the average score is the lowest, 43.6. The average distance and average score of this paper’s algorithm are in between the above two cases, i.e., the experimental results verify the effectiveness of this paper’s method. This study promotes the intelligent protection and creative inheritance of Chinese non-heritage dances through digital protection.

A Deep Vision Learning-Based Intelligent Recognition Method for Dynamic Sports Gestures

Nowadays, artificial intelligence-assisted automatic technological analyses of sports matches have received a more general demand. Among them, the effective recognition of sports actions from videos or images acts as the most fundamental issue to be solved. As sports gestures are characterized by remarkable specialty and instantaneity, it is required to develop specific effective recognition algorithms for this purpose. As a result, this work takes the case of Taekwondo sports as an example, and introduces deep vision learning to develop a specific intelligent recognition method for dynamic sports gestures. After sampling the key frames from dynamic videos, vital joint points of limbs are first obtained as the basic bottom features. Then, gesture expertise of this domain is formed to construct a multiview feature extraction work flow to better analyze complex characteristics of each gesture. Finally, the extracted feature representation is input into a recurrent neural network structure to output the discriminative results. At last, experiments are conducted for assessment, whose results show that the proposal can reach an accuracy of 95.6% and can be well suitable for the investigated problem scenario.

Vehicle Detection and Speed Estimation Using Semantic Segmentation with Low Latency

Computer vision researchers are actively studying the use of video in traffic monitoring. TrafficMonitor uses a stationary calibrated camera to automatically track and classify vehicles on roadways. In practical uses like autonomous vehicles, segmenting semantic video continues to be difficult due to high-performance standards, the high cost of convolutional neural networks (CNNs), and the significant need for low latency. An effective machine learning environment will be developed to meet the performance and latency challenges outlined above. The use of deep learning architectures like SegNet and Flownet2.0 on the CamVid dataset enables this environment to conduct pixel-wise semantic segmentation of video properties while maintaining low latency. In this work, we discuss some state-of-the-art ways to estimating the speed of vehicles, locating vehicles, and tracking objects. As a result, it is ideally suited for real-world applications since it takes advantage of both SegNet and Flownet topologies. The decision network determines whether an image frame should be processed by a segmentation network or an optical flow network based on the expected confidence score. In conjunction with adaptive scheduling of the key frame approach, this technique for decision-making can help to speed up the procedure. Using the ResNet50 SegNet model, a mean IoU of "54.27 per cent" and an average fps of "19.57" were observed. Aside from decision network and adaptive key frame sequencing, it was discovered that FlowNet2.0 increased the frames processed per second to "30.19" on GPU with such a mean IoU of "47.65%". Because the GPU was utilised "47.65%" of the time, this resulted. There has been an increase in the speed of the Video semantic segmentation network without sacrificing quality, as demonstrated by this improvement in performance.

TOWARDS AUTONOMOUS HIGH-DEFINITION MAP CONSTRUCTION

Abstract. This paper presents a GPS constrained SLAM solution, which adds reliable GPS observations as key frames to the state-of-the-art SLAM algorithm Lightweight and Ground-Optimized Lidar Odometry and Mapping (LeGO-LOAM). As the GPS has a much higher frequency than that of lidar, we first assign each lidar frame with GPS time, then every 5 seconds, a reliable GPS observation is inserted as a key frame to the pose graph, then optimizes the pose estimation and updates the old key frames. We test our method with two platforms in real driving scene, and compare its performance with LeGO-LOAM, where LeGO-LOAM cannot acquire realtime efficiency. The difference of lidar odometry of our solution with compare to RTK-GPS is less than 0.4m, with the globally referenced map can be used for relocalization.

REAL-TIME UAV 3D IMAGE POINT CLOUDS MAPPING

Abstract. This paper proposes a real-time 3D image point clouds mapping algorithm for UAVs that is capable of mapping effectively in weak GNSS environments. And a UAV mapping system is integrated with a RGB camera, an inertial measurement unit (IMU), a GNSS receiver, data transmission devices, and a DJI M300 flight platform. To achieve real-time and robust mapping, the system utilizes a visual-inertial odometry (VIO) that tightly couples GNSS, RGB image, and IMU, which provides stable state estimation information for mapping. Subsequently, a dense matching algorithm based on key frames is adopted to recover 3D mapping information with low-computational cost. Extensive experiments are conducted on our test site, demonstrating the system's ability to build maps stably, even under the effect of wind. The results compared with the trajectory reconstructed by Pix4D show that the system achieves competitive accuracy of pose estimation and is capable of real-time mapping.

RAPID AND AUTOMATED BODY MEASUREMENT OF CATTLE BASED ON STATISTICAL SHAPE MODEL

Abstract. The current methods of non-contact livestock body measurement directly deal with the low-quality point cloud data of livestock, which have low robustness and lack practicality. On the one hand, the success rate of keypoint detection for livestock body measurement is low. Due to the severe occlusion and noise in the point cloud data, body measurements of some data cannot be performed. On the other hand, the key frames need to be manually selected from the point cloud sequence during processing. Inspired by the work of 3D reconstruction based on animal statistical shape models, we implement the construction and learning of the statistical shape model of real cattle. Given the establishment of the statistical shape model of cattle, a 3D reconstruction and body measurement approach of real cattle based on low-quality point cloud data is proposed. Nine indicators are calculated and the overall estimation MAPE (Mean Absolute Percentage Error) is 10.27%. The whole process of the body measurement algorithm proposed in our paper can be extended to other quadrupeds.