Fusion Of Temporal Information Research Articles

CTIFTrack: Continuous Temporal Information Fusion for object track

In visual tracking tasks, researchers usually focus on increasing the complexity of the model or only discretely focusing on the changes in the object itself to achieve accurate recognition and tracking of the moving object. However, they often overlook the significant contribution of video-level linear temporal information fusion and continuous spatiotemporal mapping to tracking tasks. This oversight may lead to poor tracking performance or insufficient real-time ability of the model in complex scenes. Therefore, this paper proposes a real-time tracker, namely Continuous Temporal Information Fusion Tracker (CTIFTrack). The key of CTIFTrack lies in its well-designed Temporal Information Fusion (TIF) module, which cleverly performs a linear fusion of the temporal information between the (t-1)-th and the t-th frames and completes the spatiotemporal mapping. This enables the tracker to better understand the overall spatiotemporal information and contextual spatiotemporal correlations within the video, thereby having a positive impact on the tracking task. In addition, this paper also proposes the Object Template Feature Refinement (OTFR) module, which effectively captures the global information and local details of the object, and further improves the tracker’s understanding of the object features. Extensive experiments are conducted on seven benchmarks, such as LaSOT, GOT-10K, UAV123, NFS, TrackingNet, VOT2018 and OTB-100. The experimental results validate the significant contribution of the TIF module and OTFR module to the tracking task, as well as the effectiveness of CTIFTrack. It is worth noting that while maintaining excellent tracking performance, CTIFTrack also shows outstanding real-time tracking speed. On the Nvidia Tesla T4-16GB GPU, the FPS of CTIFTrack reaches 71.98. The code and demo materials will be available at https://github.com/vpsg-research/CTIFTrack.

Bi-TAM-Net framework: fault diagnosis for insulated bearing based on new noise-resistant time-series framework

Abstract Insulated bearings are extensively employed in wind turbines and other applications as essential core parts of high-power frequency control motors. However, the influence of the wind turbine structure makes it difficult to define the wind turbine insulated bearing fault signal extraction, and there is no distinctive index or qualitative formula to describe the bearing fault. In order to solve the above challenges, Bi-TAM-Net framework is developed to diagnose the insulated bearing fault signals and achieve accurate identification of bearing faults. Firstly, the temporal information feature fusion model is created by the Bi-TAM-Net framework using the time-series bearing dataset as the model data input with recursive and chain linking rules in the direction of time-series evolution. By virtue of its parallel approach and deep stacking structure, it comprehensively portrays the bearing fault data and temporally fuses the global feature information, which can effectively improve the classification performance of insulated bearing fault data. Then the self-attention mechanism is introduced into the structure of the designed temporal information fusion model for optimization, which can be modeled in sequences of arbitrary length, extracting the correlation between the features, isolating the unimportant noise information, and strengthening the extraction ability of the proposed framework for important information. Finally, in order to verify the accuracy and superiority of the developed Bi-TAM-Net framework for bearing faults, based on the same dataset, the Bi-TAM-Net framework is compared and analyzed with seven methods such as the advanced TAM-Net model, and the results show that the Bi-TAM-Net framework has better superiority. This study offers the analytical approach with engineering application value for the design and maintenance of insulated bearings used in wind turbines.

Memorizing Swin-Transformer Denoising Network for Diffusion Model

Diffusion models have garnered significant attention in the field of image generation. However, existing denoising architectures, such as U-Net, face limitations in capturing the global context, while Vision Transformers (ViTs) may struggle with local receptive fields. To address these challenges, we propose a novel Swin-Transformer-based denoising network architecture that leverages the strengths of both U-Net and ViT. Moreover, our approach integrates the k-Nearest Neighbor (kNN) based memorizing attention module into the Swin-Transformer, enabling it to effectively harness crucial contextual information from feature maps and enhance its representational capacity. Finally, we introduce an innovative hierarchical time stream embedding scheme that optimizes the incorporation of temporal cues during the denoising process. This method surpasses basic approaches like simple addition or concatenation of fixed time embeddings, facilitating a more effective fusion of temporal information. Extensive experiments conducted on four benchmark datasets demonstrate the superior performance of our proposed model compared to U-Net and ViT as denoising networks. Our model outperforms baselines on the CRC-VAL-HE-7K and CelebA datasets, achieving improved FID scores of 14.39 and 4.96, respectively, and even surpassing DiT and UViT under our experiment setting. The Memorizing Swin-Transformer architecture, coupled with the hierarchical time stream embedding, sets a new state-of-the-art in denoising diffusion models for image generation.

Semi-supervised action recognition with dynamic temporal information fusion

The most advanced semi-supervised models available are based on images for innovation, and the use of semi-supervised learning models augmented with temporal data for video-level action recognition still suffers from severe model mismatches, and the models are not sufficiently capable of capturing both local and global information about the action. Secondly the use of constant-threshold pseudo-labeling leads to low utilization of unlabeled data for difficult actions in the early stages of training, poor pseudo-labeling quality and affects recognition accuracy. To make the semi-supervised framework FixMatch more suitable for action recognition, we propose Time-Mixer and Dynamic Threshold, respectively. Time-Mixer explores complementary information between time sequences through the fusion of two-channel temporal context information. Dynamic Threshold utilizes a new core mapping function (Normal Distribution Function) to enhance pseudo-labeling quality. Extensive experiments were conducted on three action recognition datasets (Kinetics-400, UCF-101, and HMDB-51). Comprehensive experiments show that the performance of the semi-supervised model in action recognition improves considerably after using dynamic thresholding and temporal context information fusion, with a 14.4% improvement over the baseline and a 1.8% improvement over the TG (with a labeling rate of 10%) in UCF101, whereas an overall good performance is obtained for DTIF.

An Active-Matrix Synaptic Phototransistor Array for In-Sensor Spectral Processing.

The human retina perceives and preprocesses the spectral information of incident light, enabling fast image recognition and efficient chromatic adaptation. In comparison, it is reluctant to implement parallel spectral preprocessing and temporal information fusion in current complementary metal-oxide-semiconductor (CMOS) image sensors, requiring intricate circuitry, frequent data transmission, and color filters. Herein, an active-matrix synaptic phototransistor array (AMSPA) is developed based on organic/inorganic semiconductor heterostructures. The AMSPA provides wavelength-dependent, bidirectional photoresponses, enabling dynamic imaging and in-sensor spectral preprocessing functions. Specifically, near-infrared light induces inhibitory photoresponse while UV light results in exhibitory photoresponse. With rational structural design of the organic/inorganic hybrid heterostructures, the current dynamic range of phototransistor is improved to over 90 dB. Finally, a 32 × 64 AMSPA (128 pixels per inch) is demonstrated with one-switch-transistor and one-synaptic phototransistor (1-T-1-PT) structure, achieving spatial chromatic enhancement and temporal trajectory imaging. These results reveal the feasibility of AMSPA for constructing artificial vision systems.

TaReT: Temporal knowledge graph reasoning based on topology-aware dynamic relation graph and temporal fusion

Previous temporal knowledge graph (TKG) reasoning methods often focus exclusively on evolving representations. However, these methods suffer from the inadequacy of capturing the structural nuances of concurrent facts, the intricate relations in topological subgraphs, and the fusion of temporal information across timestamps. To address these challenges, this paper proposes a TKG reasoning method based on Topology-aware dynamic Relation graph and Temporal fusion (TaReT). First, TaReT proposes an innovative attention-based relational graph model, serving as a structural information encoder that captures the intricate structure of concurrent facts. Then, TaReT designs a topology-aware relational correlation unit to discern topological relation graphs of various patterns via an edge-level correlation network, yielding relation representations. Furthermore, TaReT introduces an inter-timestamp temporal information encoder which applies a dual-gate mechanism to integrate structural and relational information for temporal fusion. Finally, the temporal decoder is applied to output entity and relation predictions. Extensive experiments on four benchmark datasets establish TaReT’s superiority over leading TKG reasoning methods. On the ICEWS14 dataset, the MRR value of TaReT exceeds the reasoning baseline RE-GCN by 14.6%.

Distance Regression Enhanced with Temporal Information Fusion and Adversarial Training for Robot-Assisted Endomicroscopy.

Probe-based confocal laser endomicroscopy (pCLE) has a role in characterising tissue intraoperatively to guide tumour resection during surgery. To capture good quality pCLE data which is important for diagnosis, the probe-tissue contact needs to be maintained within a working range of micrometre scale. This can be achieved through micro-surgical robotic manipulation which requires the automatic estimation of the probe-tissue distance. In this paper, we propose a novel deep regression framework composed of the Deep Regression Generative Adversarial Network (DR-GAN) and a Sequence Attention (SA) module. The aim of DR-GAN is to train the network using an enhanced image-based supervision approach. It extents the standard generator by using a well-defined function for image generation, instead of a learnable decoder. Also, DR-GAN uses a novel learnable neural perceptual loss which combines for the first time spatial and frequency domain features. This effectively suppresses the adverse effects of noise in the pCLE data. To incorporate temporal information, we've designed the SA module which is a cross-attention module, enhanced with Radial Basis Function based encoding (SA-RBF). Furthermore, to train the regression framework, we designed a multi-step training mechanism. During inference, the trained network is used to generate data representations which are fused along time in the SA-RBF module to boost the regression stability. Our proposed network advances SOTA networks by addressing the challenge of excessive noise in the pCLE data and enhancing regression stability. It outperforms SOTA networks applied on the pCLE Regression dataset (PRD) in terms of accuracy, data quality and stability.

AMFNet: Attention-Guided Multi-Scale Fusion Network for Bi-Temporal Change Detection in Remote Sensing Images

Change detection is crucial for evaluating land use, land cover changes, and sustainable development, constituting a significant component of Earth observation tasks. The difficulty in extracting features from high-resolution images, coupled with the complexity of image content, poses challenges for traditional change detection algorithms in terms of accuracy and applicability. The recent emergence of deep learning methods has led to substantial progress in the field of change detection. However, existing frameworks often involve the simplistic integration of bi-temporal features in specific areas, lacking the fusion of temporal information and semantic details in the images. In this paper, we propose an attention-guided multi-scale fusion network (AMFNet), which effectively integrates bi-temporal image features and diverse semantics at both the encoding and decoding stages. AMFNet utilizes a unique attention-guided mechanism to dynamically adjust feature fusion, enhancing adaptability and accuracy in change detection tasks. Our method intelligently incorporates temporal information into the deep learning model, considering the temporal dependency inherent in these tasks. We decode based on an interactive feature map, which improves the model’s understanding of evolving patterns over time. Additionally, we introduce multi-level supervised training to facilitate the learning of fused features across multiple scales. In comparison with different algorithms, our proposed method achieves F1 values of 0.9079, 0.8225, and 0.8809 in the LEVIR-CD, GZ-CD, and SYSU-CD datasets, respectively. Our model outperforms the SOTA model, SAGNet, by 0.69% in terms of F1 and 1.15% in terms of IoU on the LEVIR-CD dataset, by 2.8% in terms of F1 and 1.79% in terms of IoU on the GZ-CD dataset, and by 0.54% in terms of F1 and 0.38% in terms of IoU on the SYSU-CD dataset. The method proposed in this study can be applied to various complex scenarios, establishing a change detection method with strong model generalization capabilities.

Attention-based convolutional neural network with multi-modal temporal information fusion for motor imagery EEG decoding

Convolutional neural network (CNN) has been widely applied in motor imagery (MI)-based brain computer interface (BCI) to decode electroencephalography (EEG) signals. However, due to the limited perceptual field of convolutional kernel, CNN only extracts features from local region without considering long-term dependencies for EEG decoding. Apart from long-term dependencies, multi-modal temporal information is equally important for EEG decoding because it can offer a more comprehensive understanding of the temporal dynamics of neural processes. In this paper, we propose a novel deep learning network that combines CNN with self-attention mechanism to encapsulate multi-modal temporal information and global dependencies. The network first extracts multi-modal temporal information from two distinct perspectives: average and variance. A shared self-attention module is then designed to capture global dependencies along these two feature dimensions. We further design a convolutional encoder to explore the relationship between average-pooled and variance-pooled features and fuse them into more discriminative features. Moreover, a data augmentation method called signal segmentation and recombination is proposed to improve the generalization capability of the proposed network. The experimental results on the BCI Competition IV-2a (BCIC-IV-2a) and BCI Competition IV-2b (BCIC-IV-2b) datasets show that our proposed method outperforms the state-of-the-art methods and achieves 4-class average accuracy of 85.03% on the BCIC-IV-2a dataset. The proposed method implies the effectiveness of multi-modal temporal information fusion in attention-based deep learning networks and provides a new perspective for MI-EEG decoding. The code is available at https://github.com/Ma-Xinzhi/EEG-TransNet.

Iterative Temporal-spatial Transformer-based Cardiac T1 Mapping MRI Reconstruction

The precise reconstruction of accelerated magnetic resonance imaging (MRI) brings about notable advantages, such as enhanced diagnostic precision and decreased examination costs. In contrast, traditional cardiac MRI necessitates repetitive acquisitions across multiple heartbeats, resulting in prolonged acquisition times. Significant strides have been made in accelerating MRI through deep learning-based reconstruction methods. However, these existing methods encounter certain limitations: (1) The intricate nature of heart reconstruction involving multiple complex time-series data poses a challenge in exploring nonlinear dependencies between temporal contexts. (2) Existing research often overlooks weight sharing in iterative frameworks, impeding the effective capturing of non-local information and, consequently, limiting improvements in model performance. In order to improve cardiac MRI reconstruction, we propose a novel temporal-spatial transformer with a strategy in this study. Based on the multi-level encoder and decoder transformer architecture, we conduct multi-level spatiotemporal information feature aggregation over several adjacent views, that create nonlinear dependencies among features and efficiently learn important information among adjacent cardiac temporal frames. Additionally, in order to improve contextual awareness between neighboring views, we add cross-view attention for temporal information fusion. Furthermore, we introduce an iterative strategy for training weights during the reconstruction process, which improves feature fusion in critical locations and reduces the number of computations required to calculate global feature dependencies. Extensive experiments have demonstrated the substantial superiority of this procedure over the most advanced techniques, suggesting that it has broad potential for clinical use.

The TSM-net: a new strategy for insulated bearings intelligent faults diagnosis

With the development of power semiconductor devices, pulse width modulation technology is widely used in high-power frequency conversion control motors, which significantly improves the dynamic performance of variable-speed drive system equipment. However, the high-frequency shaft voltage generated during the drive process acts on the bearing to generate high-frequency current. The damage caused by the shaft current sharply shortens the fatigue wear process of the bearing, which in turn leads to premature failure of the bearing. A high insulating ceramic coating is prepared on the outer surface and side face of the inner and outer rings of the bearing by plasma spraying. That is, an insulating protective film is formed on the outer surface of the bearing, which can effectively isolate or reduce the bearing current, prevent the occurrence of electric erosion, and prolong the service life of the variable speed drive system equipment. However, the vibration excitation generated by the variable-speed drive system equipment will cause cracks or fatigue damage to the insulating bearing, resulting in a very complex fault mechanism of the vibration signal. The fault signal characterization lacks a professional signal analysis method, especially the high-reliability, high-precision and long-life high-performance insulating bearing. There is no qualitative formula or characteristic index to explain its failure. To fill this research gap, a new strategy for optimizing the temporal information fusion model and introducing the self-attention mechanism is innovatively developed, and it is named TSM-Net model, and the first attempt is made to realize intelligent identification of insulated bearing faults. Specifically, a multi-channel insulated bearing time information fusion diagnostic model is designed, and the coarse-grained characteristics with timing law are extracted from the measured insulated bearing fault data. Then, the self-attention mechanism is introduced into the designed insulated bearing time information fusion diagnostic model to optimize, and the weight coefficient is continuously updated to calculate the correlation weight between the insulated bearing fault data and the data, so that the final decision of the TSM-Net model is more focused, so as to improve the diagnostic accuracy. Finally, comparing the proposed TSM-Net model with the current five advanced methods, it is found that the proposed TSM-Net model has good diagnostic accuracy for rail transit motor insulated bearing faults, which verifies the effectiveness and superiority of the strategy, and provides a new way for the fault diagnosis of insulated bearings of high-power inverter control motors.

Enhancing Crop Mapping Precision through Multi-Temporal Sentinel-2 Image and Spatial-Temporal Neural Networks in Northern Slopes of Tianshan Mountain

Northern Slopes of Tianshan Mountain (NSTM) in Xinjiang hold significance as a principal agricultural hub within the region’s arid zone. Accurate crop mapping across vast agricultural expanses is fundamental for intelligent crop monitoring and devising sustainable agricultural strategies. Previous studies on multi-temporal crop classification have predominantly focused on single-point pixel temporal features, often neglecting spatial data. In large-scale crop classification tasks, by using spatial information around the pixel, the contextual relationships of the crop can be obtained to reduce possible noise interference. This research introduces a multi-scale, multi-temporal classification framework centered on ConvGRU (convolutional gated recurrent unit). By leveraging the attention mechanism of the Strip Pooling Module (SPM), a multi-scale spatial feature extraction module has been designed. This module accentuates vital spatial and spectral features, enhancing the clarity of crop edges and reducing misclassifications. The temporal information fusion module integration features from various periods to bolster classification precision. Using Sentinel-2 imagery spanning May to October 2022, datasets for cotton, corn, and winter wheat of the NSTM were generated for the framework’s training and validation. The results demonstrate an impressive 93.03% accuracy for 10 m resolution crop mapping using 15-day interval, 12-band Sentinel-2 data for the three crops. This method outperforms other mainstream methods like Random Forest (RF), Long Short-Term Memory (LSTM), Transformer, and Temporal Convolutional Neural Network (TempCNN), showcasing a kappa coefficient of 0.9062, 7.52% and 2.42% improvement in Overall Accuracy compared to RF and LSTM, respectively, which demonstrate the potential of our model for large-scale crop classification tasks to enable high-resolution crop mapping on the NSTM.

A Novel 1D-Convolutional Spatial-Time Fusion Strategy for Data-Driven Fault Diagnosis of Aero-Hydraulic Pipeline Systems

Intelligent diagnosis of faults in an aero-hydraulic pipeline is important for condition monitoring of its systems. However, there are no more qualitative formulas or feature indicators to describe the faults of aero-hydraulic pipelines because of the complexity and diversity of aero-hydraulic pipeline systems, which leads to a very complex pipeline fault mechanism. In addition, although it is well known that the expression of interpretable and representable pipeline intelligent diagnosis models with pipeline fault characteristics are buried in high background noise and strong noise disturbance conditions in practical industrial scenarios, this has yet to be discussed. Inspired by the demand, this paper proposes a novel diagnosis strategy: the 1D-convolutional space-time fusion strategy for aero-engine hydraulic pipelines. Firstly, by optimizing the convolutional neural network and using it to design a one-dimensional convolutional neural network (1DCNN) with a wide input scale to expand the input field of perception, thereby obtaining more comprehensive spatial information of the pipeline data, which can effectively extract richer short sequence features. Secondly, a network of bidirectional gated recurrent Unit (Bi-GRU) is proposed, which integrates a short sequence of high-dimensional features for temporal information fusion, resulting in a certain degree of avoiding memory loss and gradient dispersion caused by the too-large step size. It is demonstrated that, for the noise signal and variable pressure signal, the fault identification accuracy approximated 95.9%, proving the proposed strategy’s robustness. By comparing with the other five methods, the proposed strategy has the ability to identify 10 different fault states in the aero-hydraulic pipeline with higher accuracy.

Multiple hypotheses based motion compensation for learned video compression

Recently, learned video compression has attracted copious research attention. However, among the existing methods, the motion used for alignment is limited to one hypothesis only, leading to inaccurate motion estimation, especially for the complicated scenes with complex movements. Motivated by multiple hypotheses philosophy in traditional video compression, we develop the multiple hypotheses based motion compensation for the learned video compression, in an effort to enhance the motion compensation efficiency by providing diverse hypotheses with efficient temporal information fusion. In particular, the multiple hypotheses module which produces multiple motions and warped features for mining sufficient temporal information, is proposed to provide various hypotheses inferences from the reference frame. To utilize these hypotheses more copiously, the hypotheses attention module is adopted by introducing the channel-wised squeeze-and-excitation layer and the multi-scale network. In addition, the context combination is employed to fuse the weighted hypotheses to generate effective contexts with powerful temporal priors. Finally, the valid contexts are used for promoting the compression efficiency by merging weighted warped features. Extensive experiments show that the proposed method can significantly improve the rate-distortion performance of learned video compression. Compared with the state-of-the-art method for end-to-end video compression, over 13% bit rate reductions on average in terms of PSNR and MS-SSIM can be achieved.

Stratification of Children with Autism Spectrum Disorder Through Fusion of Temporal Information in Eye-gaze Scan-Paths

Background: Looking pattern differences are shown to separate individuals with Autism Spectrum Disorder (ASD) and Typically Developing (TD) controls. Recent studies have shown that, in children with ASD, these patterns change with intellectual and social impairments, suggesting that patterns of social attention provide indices of clinically meaningful variation in ASD. Method: We conducted a naturalistic study of children with ASD (n = 55) and typical development (TD, n = 32). A battery of eye-tracking video stimuli was used in the study, including Activity Monitoring (AM), Social Referencing (SR), Theory of Mind (ToM), and Dyadic Bid (DB) tasks. This work reports on the feasibility of spatial and spatiotemporal scanpaths generated from eye-gaze patterns of these paradigms in stratifying ASD and TD groups. Algorithm: This article presents an approach for automatically identifying clinically meaningful information contained within the raw eye-tracking data of children with ASD and TD. The proposed mechanism utilizes combinations of eye-gaze scan-paths (spatial information), fused with temporal information and pupil velocity data and Convolutional Neural Network (CNN) for stratification of diagnosis (ASD or TD). Results: Spatial eye-gaze representations in the form of scanpaths in stratifying ASD and TD (ASD vs. TD: DNN: 74.4%) are feasible. These spatial eye-gaze features, e.g., scan-paths, are shown to be sensitive to factors mediating heterogeneity in ASD: age (ASD: 2–4 y/old vs. 10–17 y/old CNN: 80.5%), gender (Male vs. Female ASD: DNN: 78.0%) and the mixture of age and gender (5–9 y/old Male vs. 5–9 y/old Female ASD: DNN:98.8%). Limiting scan-path representations temporally increased variance in stratification performance, attesting to the importance of the temporal dimension of eye-gaze data. Spatio-Temporal scan-paths that incorporate velocity of eye movement in their images of eye-gaze are shown to outperform other feature representation methods achieving classification accuracy of 80.25%. Conclusion: The results indicate the feasibility of scan-path images to stratify ASD and TD diagnosis in children of varying ages and gender. Infusion of temporal information and velocity data improves the classification performance of our deep learning models. Such novel velocity fused spatio-temporal scan-path features are shown to be able to capture eye gaze patterns that reflect age, gender, and the mixed effect of age and gender, factors that are associated with heterogeneity in ASD and difficulty in identifying robust biomarkers for ASD.

Spatial and temporal information fusion for human action recognition via Center Boundary Balancing Multimodal Classifier

This paper proposes a novel multimodal data classifier named Center Boundary Balancing Multimodal Classifier (CBBMC) to fuse and classify the spatial and temporal descriptors for recognizing human actions from depth video sequences. CBBMC is a composite algorithm integrating feature fusion and feature classification, in which Center Boundary Balancing Projection (CBBP) is used to balance the center and boundary information of feature class spaces. In order to solve the problem of multimodal information redundancy and isolation, two feature selection and fusion schemes of CBBMC based on embedded feature selection are presented. Moreover, two new action descriptors called Gaussian Pyramid Depth Motion Images (GP-DMI) and Depth Temporal Maps (DTM) are introduced to capture the multi-scale spatial and fine-grained temporal information of human activities. Finally, we present an effective spatial and temporal information fusion framework based on CBBMC for human action recognition. In order to evaluate the performance of the proposed approach, extensive experiments are conducted. The proposed method achieved impressive results on four benchmark datasets, namely MSR Action3D (96.33%), UTD-MHAD (94.41%), DHA (95.65%), and NTU RGB+D (83.31% cross-subject and 87.66% cross-view), even though only the depth modality was used. The experimental results demonstrate the effectiveness of our method.

Deep Learning-Based Data Augmentation and Model Fusion for Automatic Arrhythmia Identification and Classification Algorithms.

Automated ECG-based arrhythmia detection is critical for early cardiac disease prevention and diagnosis. Recently, deep learning algorithms have been widely applied for arrhythmia detection with great success. However, the lack of labeled ECG data and low classification accuracy can have a significant impact on the overall effectiveness of a classification algorithm. In order to better apply deep learning methods to arrhythmia classification, in this study, feature extraction and classification strategy based on generative adversarial network data augmentation and model fusion are proposed to address these problems. First, the arrhythmia sparse data is augmented by generative adversarial networks. Then, aiming at the identification of different types of arrhythmias in long-term ECG, a spatial information fusion model based on ResNet and a temporal information fusion model based on BiLSTM are proposed. The model effectively fuses the location information of the nearest neighbors through the local feature extraction part of the generated ECG feature map and obtains the correlation of the global features by autonomous learning in multiple spaces through the BiLSTM network in the part of the global feature extraction. In addition, an attention mechanism is introduced to enhance the features of arrhythmia-type signal segments, and this mechanism can effectively focus on the extraction of key information to form a feature vector for final classification. Finally, it is validated by the enhanced MIT-BIH arrhythmia database. The experimental results demonstrate that the proposed classification technique enhances arrhythmia diagnostic accuracy by 99.4%, and the algorithm has high recognition performance and clinical value.

TERQA: Question answering over knowledge graph considering precise dependencies of temporal information on vectors

Time questions involve explicit and implicit constraints as well as complex time interval interactions, making them critical criteria for measuring the effect of knowledge base question answering. Although attention to temporal questions has spurred the development of temporal knowledge graphs, existing studies have focused on the simple splicing and fusion of temporal information with question or knowledge base embeddings, losing sight of the hidden interaction features between temporal information and embedded vectors. In this paper, we proposed TERQA, a temporal knowledge base question-answering approach to explore precise spatial dependencies between temporal information and embedded vectors. The exploration of the deep dependency between time and embedded vectors was divided into two stages. In the first stage, the Transformer model of depth extraction was employed to extract richer features from questions and the representation was enhanced with temporal information; in the second stage, high-level capsules were adopted to extract the low-level vector features for detailed pose determination, allowing a more precise deep dependency of temporal facts on embedded vectors. We conducted an experiment using two temporal question answering datasets, TempQuestions and CronQuestions, and the results showed that accuracy for TERQA improved 11.3% from baseline on the dataset TempQuestions with higher annotated information. Additionally, the adapted TERQA also showed varying degrees of improvements over the baseline in the larger but simply annotated dataset CronQuestions.

Temporal Group Fusion Network for Deep Video Inpainting

Video inpainting is a task of synthesizing spatio-temporal coherent content in missing regions of the given video sequence, which has recently drawn increasing attention. To utilize the temporal information across frames, most recent deep learning-based methods align reference frames to target frame firstly with explicit or implicit motion estimation and then integrate the information from the aligned frames. However, their performance relies heavily on the accuracy of frame-to-frame alignment. To alleviate the above problem, in this paper, a novel Temporal Group Fusion Network (TGF-Net) is proposed to effectively integrate temporal information through a two-stage fusion strategy. Specifically, the input frames are reorganized into different groups, where each group is followed by an intra-group fusion module to integrate information within the group. Different groups provide complementary information for the missing region. A temporal attention model is further designed to adaptively integrate the information across groups. Such a temporal information fusion way gets rid of the dependence on alignment operations, greatly improving the visual quality and temporal consistency of the inpainted results. In addition, a coarse alignment model is introduced at the beginning of the network to handle videos with large motion. Extensive experiments on DAVIS and Youtube-VOS datasets demonstrate the superiority of our proposed method in terms of PSNR/SSIM values, visual quality and temporal consistency, respectively.

Attention Mechanism Based on Improved Spatial-Temporal Convolutional Neural Networks for Traffic Police Gesture Recognition

Human action recognition has attracted extensive research efforts in recent years, in which traffic police gesture recognition is important for self-driving vehicles. One of the crucial challenges in this task is how to find a representation method based on spatial-temporal features. However, existing methods performed poorly in spatial and temporal information fusion, and how to extract features of traffic police gestures has not been well researched. This paper proposes an attention mechanism based on the improved spatial-temporal convolutional neural network (AMSTCNN) for traffic police gesture recognition. This method focuses on the action part of traffic police and uses the correlation between spatial and temporal features to recognize traffic police gestures, so as to ensure that traffic police gesture information is not lost. Specifically, AMSTCNN integrates spatial and temporal information, uses weight matching to pay more attention to the region where human action occurs, and extracts region proposals of the image. Finally, we use Softmax to classify actions after spatial-temporal feature fusion. AMSTCNN can strongly make use of the spatial-temporal information of videos and select effective features to reduce computation. Experiments on AVA and the Chinese traffic police gesture datasets show that our method is superior to several state-of-the-art methods.