Feature Representation Ability Research Articles

A Parallel Convolutional Network Based on Spiking Neural Systems.

Deep convolutional neural networks have shown advanced performance in accurately segmenting images. In this paper, an SNP-like convolutional neuron structure is introduced, abstracted from the nonlinear mechanism in nonlinear spiking neural P (NSNP) systems. Then, a U-shaped convolutional neural network named SNP-like parallel-convolutional network, or SPC-Net, is constructed for segmentation tasks. The dual-convolution concatenate (DCC) and dual-convolution addition (DCA) network blocks are designed, respectively, in the encoder and decoder stages. The two blocks employ parallel convolution with different kernel sizes to improve feature representation ability and make full use of spatial detail information. Meanwhile, different feature fusion strategies are used to fuse their features to achieve feature complementarity and augmentation. Furthermore, a dual-scale pooling (DSP) module in the bottleneck is designed to improve the feature extraction capability, which can extract multi-scale contextual information and reduce information loss while extracting salient features. The SPC-Net is applied in medical image segmentation tasks and is compared with several recent segmentation methods on the GlaS and CRAG datasets. The proposed SPC-Net achieves 90.77% DICE coefficient, 83.76% IoU score and 83.93% F1 score, 86.33% ObjDice coefficient, 135.60 Obj-Hausdorff distance, respectively. The experimental results show that the proposed model can achieve good segmentation performance.

CD-MAE: Contrastive Dual-Masked Autoencoder Pre-Training Model for PCB CT Image Element Segmentation

Element detection is an important step in the process of the non-destructive testing of printed circuit boards (PCB) based on computed tomography (CT). Compared with the traditional manual detection method, the image semantic segmentation method based on deep learning greatly improves efficiency and accuracy. However, semantic segmentation models often require a large amount of data for supervised training to generalize better model performance. Unlike natural images, the PCB CT image annotation task is more time-consuming and laborious than the semantic segmentation task. In order to reduce the cost of labeling and improve the ability of the model to utilize unlabeled data, unsupervised pre-training is a very reasonable and necessary choice. The masked image reconstruction model represented by a masked autoencoder is pre-trained on the unlabeled data, learning a strong feature representation ability by recovering the masked image, and shows a good generalization ability in various downstream tasks. In the PCB CT image element segmentation task, considering the characteristics of the image, it is necessary to use a model with strong feature robustness in the pre-training stage to realize the representation learning on a large number of unlabeled PCB CT images. Based on the above purposes, we proposed a contrastive dual-masked autoencoder (CD-MAE) pre-training model, which can learn more robust feature representation on unlabeled PCB CT images. Our experiments show that the CD-MAE outperforms the baseline model and fully supervised models in the PCB CT element segmentation task.

Multi-axis interactive multidimensional attention network for vehicle re-identification

Learning fine-grained discriminative information is essential to address the challenges of small inter-class differences and large intra-class differences in vehicle re-identification (Re-ID). Attentional mechanism is often used to capture important global information in images rather than fine-grained discriminative information. Studies have shown that the multi-axis interaction of information can enhance the feature representation ability of networks. This paper explores how to use the multi-axis interaction of information to facilitate more effective learning of attention and how to capture important detailed information in local regions. We propose a multi-axis interactive multidimensional attention network (MIMA-Net) for vehicle Re-ID. The network allows information to interact on multiple axes and calibrates the weight distribution of features from multiple dimensions to learn subtle discriminative information in vehicle parts/regions. The window-channel attention module (W-CAM) in MIMA-Net facilitates the learning of channel attention by interacting first across locations and then across channels, while the channel group-spatial attention module (CG-SAM) facilitates the learning of spatial attention by interacting first across channels and then across locations. These two modules perform window partitioning in a priori manner and channel semantic aggregation in an adaptive manner to learn discriminative semantic features in parts, respectively. These two approaches complement each other to strengthen the feature representation ability of MIMA-Net. Extensive experiments on three large public datasets, VeRi-776, VehicleID, and VERI-Wild, verify the effectiveness of our MIMA-Net and show that our method achieves state-of-the-art performance.

DI-VTR: Dual inter-modal interaction model for video-text retrieval

Video-text retrieval is a challenging task for multimodal information processing due to the semantic gap between different modalities. However, most existing methods do not fully mine the intra-modal interactions, as with the temporal correlation of video frames, which results in poor matching performance. Additionally, the imbalanced semantic information between videos and texts also leads to difficulty in the alignment of the two modalities. To this end, we propose a dual inter-modal interaction network for video-text retrieval, i.e., DI-VTR. To learn the intra-modal interaction of video frames, we design a contextual-related video encoder to obtain more fine-grained content-oriented video representations. We also propose a dual inter-modal interaction module to accomplish accurate multilingual alignment between the video and text modalities by introducing multilingual text to improve the representation ability of text semantic features. Extensive experimental results on commonly-used video-text retrieval datasets, including MSR-VTT, MSVD and VATEX, show that the proposed method achieves significantly improved performance compared with state-of-the-art methods.

A Practical Deep Learning Architecture for Large-Area Solid Wastes Monitoring Based on UAV Imagery

The development of global urbanization has brought about a significant amount of solid waste. These untreated wastes may be dumped in any corner, causing serious pollution to the environment. Thus, it is necessary to accurately obtain their distribution locations and detailed edge information. In this study, a practical deep learning network for recognizing solid waste piles over extensive areas using unmanned aerial vehicle (UAV) imagery has been proposed and verified. Firstly, a high-resolution dataset serving to solid waste detection was created based on UAV aerial data. Then, a dual-branch solid waste semantic segmentation model was constructed to address the characteristics of the integration of solid waste distribution with the environment and the irregular edge morphology. The Context feature branch is responsible for extracting high-level semantic features, while the Spatial feature branch is designed to capture fine-grained spatial details. After information fusion, the model obtained more comprehensive feature representation and segmentation ability. The effectiveness of the improvement was verified through ablation experiments and compared with 13 commonly used semantic segmentation models, demonstrating the advantages of the method in solid waste segmentation tasks, with an overall accuracy of over 94%, and a recall rate of 88.6%—much better than the best performing baselines. Finally, a spatial distribution map of solid waste over Jiaxing district, China was generated by the model inference, which assisted the environmental protection department in completing environmental management. The proposed method provides a feasible approach for the accurately monitoring of solid waste, so as to provide policy support for environmental protection.

S3-Net: A Self-Supervised Dual-Stream Network for Radiology Report Generation.

Intelligent medicine is eager to automatically generate radiology reports to ease the tedious work of radiologists. Previous researches mainly focused on the text generation with encoder-decoder structure, while CNN networks for visual features ignored the long-range dependencies correlated with textual information. Besides, few studies exploit cross-modal mappings to promote radiology report generation. To alleviate the above problems, we propose a novel end-to-end radiology report generation model dubbed Self-Supervised dual-Stream Network (S3-Net). Specifically, a Dual-Stream Visual Feature Extractor (DSVFE) composed of ResNet and SwinTransformer is proposed to capture more abundant and effective visual features, where the former focuses on local response and the latter explores long-range dependencies. Then, we introduced the Fusion Alignment Module (FAM) to fuse the dual-stream visual features and facilitate alignment between visual features and text features. Furthermore, the Self-Supervised Learning with Mask(SSLM) is introduced to further enhance the visual feature representation ability. Experimental results on two mainstream radiology reporting datasets (IU X-ray and MIMIC-CXR) show that our proposed approach outperforms previous models in terms of language generation metrics.

Hyperspectral image classification based on local feature decoupling and hybrid attention SpectralFormer network

ABSTRACT Classification of hyperspectral image (HSI) is an essential step in various remote sensing tasks. Currently, transformer-based models have made significant contributions to HSI classification due to their remarkable ability to model the relation among long-range sequence data. However, HSI data exhibit strong nonlinear coupling and strong correlation, leading to a decrease in the classification performance of transformer-based models. The SpectralFormer network has achieved excellent performance in HSI classification, but its cross-layer adaptive fusion mechanism cannot effectively highlight critical information, affecting the representation ability of spectral-spatial features. To address the above issues, this paper proposes an HSI classification method based on local feature decoupling and SpectralFormer with a hybrid attention module (HAMSF) network. Specifically, this paper first uses the histogram of oriented gradient algorithm to preprocess HSI coupled data and achieve preliminary nonlinear decoupling of HSI data. Secondly, the hybrid attention module is embedded in a cross-layer adaptive fusion mechanism, generating refined neighbouring information before each information fusion to improve the spectral-spatial feature representation ability. Finally, the processed data are input into the HAMSF network for the final decoupling and classification of HSI. The experiment is carried out on three benchmark hyperspectral datasets, and the experimental results confirm that the proposed method has better classification performance compared to several transformer-based and classical HSI classification methods.

Low-Dose CT Image Super-resolution Network with Noise Inhibition Based on Feedback Feature Distillation Mechanism.

Low-dose computed tomography (LDCT) has been widely used in medical diagnosis. In practice, doctors often zoom in on LDCT slices for clearer lesions and issues, while, a simple zooming operation fails to suppress low-dose artifacts, leading to distorted details. Therefore, numerous LDCT super-resolution (SR) methods have been proposed to promote the quality of zooming without the increase of the dose in CT scanning. However, there are still some drawbacks that need to be addressed in existing methods. First, the region of interest (ROI) is not emphasized due to the lack of guidance in the reconstruction process. Second, the convolutional blocks extracting fix-resolution features fail to concentrate on the essential multi-scale features. Third, a single SR head cannot suppress the residual artifacts. To address these issues, we propose an LDCT CT joint SR and denoising reconstruction network. Our proposed network consists of global dual-guidance attention fusion modules (GDAFMs) and multi-scale anastomosis blocks (MABs). The GDAFM directs the network to focus on ROI by fusing the extra mask guidance and average CT image guidance, while the MAB introduces hierarchical features through anastomosis connections to leverage multi-scale features and promote the feature representation ability. To suppress radial residual artifacts, we optimize our network using the feedback feature distillation mechanism (FFDM) which shares the backbone to learn features corresponding to the denoising task. We apply the proposed method to the 3D-IRCADB and PANCREAS datasets to evaluate its ability on LDCT image SR reconstruction. The experimental results compared with state-of-the-art methods illustrate the superiority of our approach with respect to peak signal-to-noise (PSNR), structural similarity (SSIM), and qualitative observations. Our proposed LDCT joint SR and denoising reconstruction network has been extensively evaluated through ablation, quantitative, and qualitative experiments. The results demonstrate that our method can recover noise-free and detail-sharp images, resulting in better reconstruction results. Code is available at https://github.com/neu-szy/ldct_sr_dn_w_ffdm .

LM-DeeplabV3+: A Lightweight Image Segmentation Algorithm Based on Multi-Scale Feature Interaction

Street-view images can help us to better understand the city environment and its potential characteristics. With the development of computer vision and deep learning, the technology of semantic segmentation algorithms has become more mature. However, DeeplabV3+, which is commonly used in semantic segmentation, has shortcomings such as a large number of parameters, high requirements for computing resources, and easy loss of detailed information. Therefore, this paper proposes LM-DeeplabV3+, which aims to greatly reduce the parameters and computations of the model while ensuring segmentation accuracy. Firstly, the lightweight network MobileNetV2 is selected as the backbone network, and the ECA attention mechanism is introduced after MobileNetV2 extracts shallow features to improve the ability of feature representation; secondly, the ASPP module is improved, and on this basis, the EPSA attention mechanism is introduced to achieve cross-dimensional channel attention and important feature interaction; thirdly, a loss function named CL loss is designed to balance the training offset of multiple categories and better indicate the segmentation quality. This paper conducted experimental verification on the Cityspaces dataset, and the results showed that the mIoU reached 74.9%, which was an improvement of 3.56% compared to DeeplabV3+; and the mPA reached 83.01%, which was an improvement of 2.53% compared to DeeplabV3+.

A Deep Learning Method for Building Extraction from Remote Sensing Images by Fuzing Local and Global Features

As important disaster-bearing bodies, buildings are the focus of attention in seismic disaster risk assessment and emergency rescue. It is of great practical significance to extract buildings quickly and accurately with complex textures and variable scales and shapes from high-resolution remote sensing images. We proposed an improved TransUnet model based on multiscale grouped convolution and attention named MATUnet to retain more local detail features and enhance the representation ability of global features, while reducing the network parameters. We designed the multiscale grouped convolutional feature extraction module with attention (GAM) to enhance the representation of detailed features. The convolutional positional encoding module (PEG) was added to redetermine the number of transformer, it solved the problem of local feature information loss and the difficulty of convergence of the network. The channel attention module (CAM) of the decoder enhanced the salient information of the features and solved the problem of information redundancy after feature fusion. We experimented through MATUnet on the WHU building dataset and Massachusetts dataset. MATUnet achieved the best IOU results of 92.14% and 83.22%, respectively, and achieved better than the other generalized and state-of-the-art networks under the same conditions. We also have achieved good segmentation results on the GF2 Xichang building dataset.

Research on surface defect detection algorithm of pipeline weld based on YOLOv7

Aiming at the problems of low target detection accuracy and high leakage rate of the current traditional weld surface defect detection methods and existing detection models, an improved YOLOv7 pipeline weld surface defect detection model is proposed to improve detection results. In the improved model, a Le-HorBlock module is designed, and it is introduced into the back of fourth CBS module of the backbone network, which preserves the characteristics of high-order information by realizing second-order spatial interaction, thus enhancing the ability of the network to extract features in weld defect images. The coordinate attention (CoordAtt) block is introduced to enhance the representation ability of target features, suppress interference. The CIoU loss function in YOLOv7 network model is replaced by the SIoU, so as to optimize the loss function, reduce the freedom of the loss function, and accelerate convergence. And a new large-scale pipeline weld surface defect dataset containing 2000 images of pipeline welds with weld defects is used in the proposed model. In the experimental comparison, the improved YOLOv7 network model has greatly improved the missed detection rate compared with the original network. The experimental results show that the improved YOLOv7 network model mAP@80.5 can reach 78.6%, which is 15.9% higher than the original model, and the detection effect is better than the original network and other classical target detection networks.

Dual-Branch Cross-Attention Network for Micro-Expression Recognition with Transformer Variants

A micro-expression (ME), as a spontaneous facial expression, usually occurs instantaneously and is difficult to disguise after an emotion-evoking event. Numerous convolutional neural network (CNN)-based models have been widely explored to recognize MEs for their strong local feature representation ability on images. However, the main drawback of the current methods is their inability to fully extracting holistic contextual information from ME images. To achieve efficient ME learning representation from diverse perspectives, this paper uses Transformer variants as the main backbone and the dual-branch architecture as the main framework to extract meaningful multi-modal contextual features for ME recognition (MER). The first branch leverages an optical flow operator to facilitate the motion information extraction between ME sequences, and the corresponding optical flow maps are fed into the Swin Transformer to acquire motion–spatial representation. The second branch directly sends the apex frame in one ME clip to Mobile ViT (Vision Transformer), which can capture the local–global features of MEs. More importantly, to achieve the optimal feature stream fusion, a CAB (cross attention block) is designed to interact the feature extracted by each branch for adaptive learning fusion. The extensive experimental comparisons on three publicly available ME benchmarks show that the proposed method outperforms the existing MER methods and achieves an accuracy of 81.6% on the combined database.

EMAT: Efficient feature fusion network for visual tracking via optimized multi-head attention

The tracking methods based on Transformer have shown great potential in visual tracking and achieved significant tracking performance. The traditional transformer based feature fusion network divides a whole feature map into multiple image patches as its inputs, and then directly processes them in parallel, which will occupy a lot of computing resources and affect the computing efficiency of multi-head attention. In this paper, we design a novel feature fusion network with optimized multi-head attention in encoder and decoder architecture based on Transformer. The designed feature fusion network preprocess the input features and change the calculations of multi-head attention by using both the efficient multi-head self-attention module and efficient multi-head spatial reduction attention module. The two modules can reduce the influence of irrelevant background information, enhance the representation ability of template features and search region features, and greatly reduce the computational complexity. We propose a novel Transformer tracking method (named EMAT) based on the designed feature fusion network. The proposed EMAT is evaluated on seven challenging tracking benchmarks to demonstrate its superiority, including LaSOT, GOT-10k, TrackingNet, UAV123, VOT2018, NfS and VOT-RGBT2019. The proposed tracker achieves well tracking performance, and obtains precision score of 89.0% on UAV123, AUC score of 64.6% on LaSOT, EAO score of 34.8% on VOT-RGBT2019, which outperforms most advanced trackers. EMAT runs at a real-time speed of about 35 FPS during tracking.

Deep convolutional dictionary learning network for sparse view CT reconstruction with a group sparse prior

Purpose Numerous techniques based on deep learning have been utilized in sparse view computed tomography (CT) imaging. Nevertheless, the majority of techniques are instinctively constructed utilizing state-of-the-art opaque convolutional neural networks (CNNs) and lack interpretability. Moreover, CNNs tend to focus on local receptive fields and neglect nonlocal self-similarity prior information. Obtaining diagnostically valuable images from sparsely sampled projections is a challenging and ill-posed task.Method To address this issue, we propose a unique and understandable model named DCDL-GS for sparse view CT imaging. This model relies on a network comprised of convolutional dictionary learning and a nonlocal group sparse prior. To enhance the quality of image reconstruction, we utilize a neural network in conjunction with a statistical iterative reconstruction framework and perform a set number of iterations. Inspired by group sparsity priors, we adopt a novel group thresholding operation to improve the feature representation and constraint ability and obtain a theoretical interpretation. Furthermore, our DCDL-GS model incorporates filtered backprojection (FBP) reconstruction, fast sliding window nonlocal self-similarity operations, and a lightweight and interpretable convolutional dictionary learning network to enhance the applicability of the model.Results The efficiency of our proposed DCDL-GS model in preserving edges and recovering features is demonstrated by the visual results obtained on the LDCT-P and UIH datasets. Compared to the results of the most advanced techniques, the quantitative results are enhanced, with increases of 0.6-0.8 dB for the peak signal-to-noise ratio (PSNR), 0.005-0.01 for the structural similarity index measure (SSIM), and 1-1.3 for the regulated Fréchet inception distance (rFID) on the test dataset. The quantitative results also show the effectiveness of our proposed deep convolution iterative reconstruction module and nonlocal group sparse prior.Conclusion In this paper, we create a consolidated and enhanced mathematical model by integrating projection data and prior knowledge of images into a deep iterative model. The model is more practical and interpretable than existing approaches. The results from the experiment show that the proposed model performs well in comparison to the others.

Improved YOLOv7 model for insulator defect detection

<abstract> <p>Insulators are crucial insulation components and structural supports in power grids, playing a vital role in the transmission lines. Due to temperature fluctuations, internal stress, or damage from hail, insulators are prone to injury. Automatic detection of damaged insulators faces challenges such as diverse types, small defect targets, and complex backgrounds and shapes. Most research for detecting insulator defects has focused on a single defect type or a specific material. However, the insulators in the grid's transmission lines have different colors and materials. Various insulator defects coexist, and the existing methods have difficulty meeting the practical application requirements. Current methods suffer from low detection accuracy and mAP0.5 cannot meet application requirements. This paper proposes an improved you only look once version 7 (YOLOv7) model for multi-type insulator defect detection. First, our model replaces the spatial pyramid pooling cross stage partial network (SPPCSPC) module with the receptive filed block (RFB) module to enhance the network's feature extraction capability. Second, a coordinate attention (CA) mechanism is introduced into the head part to enhance the network's feature representation ability and to improve detection accuracy. Third, a wise intersection over union (WIoU) loss function is employed to address the low-quality samples hindering model generalization during training, thereby improving the model's overall performance. The experimental results indicate that the proposed model exhibits enhancements across various performance metrics. Specifically, there is a 1.6% advancement in mAP_0.5, a corresponding 1.6% enhancement in mAP_0.5:0.95, a 1.3% elevation in precision, and a 1% increase in recall. Moreover, the model achieves parameter reduction by 3.2 million, leading to a decrease of 2.5 GFLOPS in computational cost. Notably, there is also an improvement of 2.81 milliseconds in single-image detection speed. This improved model can detect insulator defects for diverse materials, color insulators, and partial damage shapes in complex backgrounds.</p> </abstract>

Multi-Modal Fusion Based on Depth Adaptive Mechanism for 3D Object Detection

Lidars and cameras are critical sensors for 3D object detection in autonomous driving. Despite the increasing popularity of sensor fusion in this field, accurate and robust fusion methods are still under exploration due to non-homogenous representations. In this paper, we find that the complementary roles of point clouds and images vary with depth. An important reason is that the point cloud appearance changes significantly with increasing distance from the Lidar, while the image's edge, color, and texture information are not sensitive to depth. To address this, we propose a fusion module based on the Depth Attention Mechanism (DAM), which mainly consists of two operations: gated feature generation and point cloud division. The former adaptively learns the importance of bimodal features without additional annotations, while the latter divides point clouds to achieve differential fusion of multi-modal features at different depths. This fusion module can enhance the representation ability of original features for different point sets and provide more comprehensive features by using the dual splicing strategy of concatenation and index connection. Additionally, considering point density as a feature and its negative correlation with depth, we build an Adaptive Threshold Generation Network (ATGN) to generate the depth threshold by extracting density information, which can divide point clouds more reasonably. Experiments on the KITTI dataset demonstrate the effectiveness and competitiveness of our proposed models.

A New Siamese Heterogeneous Convolutional Neural Networks Based on Attention Mechanism and Feature Pyramid.

Accuracy and speed are the most important indexes for evaluating many object tracking algorithms. However, when constructing a deep fully convolutional neural network (CNN), the use of deep network feature tracking will cause tracking drift due to the effects of convolution padding, receptive field (RF), and overall network step size. The speed of the tracker will also decrease. This article proposes a fully convolutional siamese network object tracking algorithm that combines the attention mechanism with the feature pyramid network (FPN), and uses heterogeneous convolution kernels to reduce the amount of calculations (FLOPs) and parameters. The tracker first uses a new fully CNN to extract image features, and introduces a channel attention mechanism in the feature extraction process to improve the representation ability of convolutional features. Then use the FPN to fuse the convolutional features of high and low layers, learn the similarity of the fused features, and train the fully CNNs. Finally, the heterogeneous convolutional kernel is used to replace the standard convolution kernel to improve the speed of the algorithm, thereby making up for the efficiency loss caused by the feature pyramid model. In this article, the tracker is experimentally verified and analyzed on the VOT-2017, VOT-2018, OTB-2013, and OTB-2015 datasets. The results show that our tracker has achieved better results than the state-of-the-art trackers.

An Efficient Single Image De-raining Model with Decoupled Deep Networks.

Single image de-raining is an emerging paradigm for many outdoor computer vision applications since rain streaks can significantly degrade the visibility and render the function compromised. The introduction of deep learning (DL) has brought about substantial advancement on de-raining methods. However, most existing DL-based methods use single homogeneous network architecture to generate de-rained images in a general image restoration manner, ignoring the discrepancy between rain location detection and rain intensity estimation. We find that this discrepancy would cause feature interference and representation ability degradation problems which significantly affect de-raining performance. In this paper, we propose a novel heterogeneous de-raining architecture aiming to decouple rain location detection and rain intensity estimation (DLINet). For these two subtasks, we provide dedicated network structures according to their differential properties to meet their respective performance requirements. To coordinate the decoupled subnetworks, we develop a high-order collaborative network learning the dynamic inter-layer interactions between rain location and intensity. To effectively supervise the decoupled subnetworks during training, we propose a novel training strategy that imposes task-oriented supervision using the label learned via joint training. Extensive experiments on synthetic datasets and real-world rainy scenes demonstrate that the proposed method has great advantages over existing state-of-the-art methods.

SFWN: A Novel Semi-Supervised Feature Weighted Neural Network for Gene Data Feature Learning and Mining With Graph Modeling.

Gene expression data can serve for analyzing the genes with changed expressions, the correlation between genes and the influence of different circumstance on gene activities. However, labeling a large number of gene expression data is laborious and time-consuming. The insufficient labeled data pose a challenge to construct the deep learning model. Currently, some graph neural networks (GNN) based on semi-supervised learning mechanism only focus on the feature space and sample space of gene expression data, possibly affecting the accuracy. This article puts forward a novel semi-supervised graph neural network model (SFWN). Firstly, we use the external knowledge of gene expression data for constructing a feature graph, a similarity kernel, and a sample graph for the first time. Later, a novel semi-supervised learning algorithm (SGA) is proposed to extract the data relationship and obtain the global sample structure better. A graph sparse module (SGCN) is also proposed to process sparse representation with gene expression data classification. To overcome the over smoothing problem, a new feature calculation method based on two spaces is proposed to feature representation analysis and calculation in this model. According to a lot of experiments and ablation studies conducted on several public datasets, SFWN exhibits a better effect and is superior to the state-of-the-art approaches (the accuracy and F1-Score are 0.9993 and 0.9899, respectively). Experimental results showed that the proposed SFWN model has strong gene expression feature learning and representation ability, and may provide a new insight and tool for relevant disease diagnosis and clinic practice.

VT-3DCapsNet: Visual tempos 3D-Capsule network for video-based facial expression recognition.

Facial expression recognition(FER) is a hot topic in computer vision, especially as deep learning based methods are gaining traction in this field. However, traditional convolutional neural networks (CNN) ignore the relative position relationship of key facial features (mouth, eyebrows, eyes, etc.) due to changes of facial expressions in real-world environments such as rotation, displacement or partial occlusion. In addition, most of the works in the literature do not take visual tempos into account when recognizing facial expressions that possess higher similarities. To address these issues, we propose a visual tempos 3D-CapsNet framework(VT-3DCapsNet). First, we propose 3D-CapsNet model for emotion recognition, in which we introduced improved 3D-ResNet architecture that integrated with AU-perceived attention module to enhance the ability of feature representation of capsule network, through expressing deeper hierarchical spatiotemporal features and extracting latent information (position, size, orientation) in key facial areas. Furthermore, we propose the temporal pyramid network(TPN)-based expression recognition module(TPN-ERM), which can learn high-level facial motion features from video frames to model differences in visual tempos, further improving the recognition accuracy of 3D-CapsNet. Extensive experiments are conducted on extended Kohn-Kanada (CK+) database and Acted Facial Expression in Wild (AFEW) database. The results demonstrate competitive performance of our approach compared with other state-of-the-art methods.