Discriminative Spatial Features Research Articles

OA-GAN: organ-aware generative adversarial network for synthesizing contrast-enhanced medical images

Contrast-enhanced computed tomography (CE-CT) images are vital for clinical diagnosis of focal liver lesions (FLLs). However, the use of CE-CT images imposes a significant burden on patients due to the injection of contrast agents and extended shooting. Deep learning-based image synthesis models offer a promising solution that synthesizes CE-CT images from non-contrasted CT (NC-CT) images. Unlike natural images, medical image synthesis requires a specific focus on certain organs or localized regions to ensure accurate diagnosis. Determining how to effectively emphasize target organs poses a challenging issue in medical image synthesis. To solve this challenge, we present a novel CE-CT image synthesis model called, Organ-Aware Generative Adversarial Network (OA-GAN). The OA-GAN comprises an organ-aware (OA) network and a dual decoder-based generator. First, the OA network learns the most discriminative spatial features about the target organ (i.e. liver) by utilizing the ground truth organ mask as localization cues. Subsequently, NC-CT image and captured feature are fed into the dual decoder-based generator, which employs a local and global decoder network to simultaneously synthesize the organ and entire CECT image. Moreover, the semantic information extracted from the local decoder is transferred to the global decoder to facilitate better reconstruction of the organ in entire CE-CT image. The qualitative and quantitative evaluation on a CE-CT dataset demonstrates that the OA-GAN outperforms state-of-the-art approaches for synthesizing two types of CE-CT images such as arterial phase and portal venous phase. Additionally, subjective evaluations by expert radiologists and a deep learning-based FLLs classification also affirm that CE-CT images synthesized from the OA-GAN exhibit a remarkable resemblance to real CE-CT images.

Video Hashing with Tensor Robust PCA and Histogram of Optical Flow for Copy Detection

Abstract This paper proposes a novel video hashing with tensor robust Principal Component Analysis (PCA) and Histogram of Optical Flow (HOF) for copy detection. In the proposed hashing, a video is divided into some video groups. For each video group, a low-rank secondary frame is constructed from the low-rank component decomposed by applying tensor robust PCA to the video group. Since the low-rank component can well indicate spatial-temporal intrinsic structure of the video group and it is slightly disturbed by digital operations, feature extraction from the low-rank secondary frames is discriminative and stable. Next, spatial features and temporal features are extracted from low-rank secondary frames by Charlier moments and HOF, respectively. Since the Charlier moments are robust to geometric transform and they can efficiently distinguish video frames with different contents, the use of Charlier moments can make robust and discriminative spatial features. As the HOF can measure the distribution of motion information between frames, the temporal features formed by HOFs can provide good discrimination. Hash is ultimately determined by quantizing the spatial and temporal features and concatenating the quantized results. Numerous experiments on open video datasets indicate that the proposed hashing is superior to some hashing baseline schemes in terms of classification and copy detection.

Learning discriminative feature representation with pixel-level supervision for forest smoke recognition

Existing vision-based smoke recognition methods still face the issues of low detection rates and high false alarm rates in complex scenes. One reason is that they label light smoke and heavy smoke as the same value, which ignores the differences in multiple attribute information involved in the smoke imaging process. To solve this issue, this paper presents a pixel-level supervision neural network (PSNet) to learn discriminative feature representations for forest smoke recognition. First, the pixel-level supervision information, including the background component, smoke component, fusion ratio, and class information, is cooperatively considered to effectively guide the model training process. To avoid negative transfer caused by the asynchronous optimization of shared layer parameters and achieve synchronous minimization of each loss term, a regularization term based on the smoke imaging principle and a weight dynamic updating method are proposed to balance the weight coefficients of different loss terms. Second, a detail-difference-aware module (DDAM) based on a detail-difference-aware block (DDAB) and a spatial attention block (SAB) is proposed to distinguish smoke and smoke-like targets by fusing xy-shared convolution and z-shared convolution, which adaptively allocates the weights over different positions to prioritize the most informative visual elements in the spatial domain. Third, an attention-based feature separation module (AFSM) is proposed to relieve mutual interference in extracting background features and smoke features by designing component interaction attention (CIA), background component attention (BCA), smoke component attention (SCA), and enhanced residual blocks (ERBs), which can guide the interaction and separation process of background information and smoke information to enhance the discriminative spatial features and suppress interference features. ERB effectively eliminates noise and enhances smoke edge information based on median filters. Finally, to further enhance the feature representation capability, a multiconnection aggregation method (MCAM) is proposed by fully aggregating local and global features simultaneously. Extensive experiments show that our method achieves better performance than existing smoke recognition methods.Extensive experiments show that our PSNet achieves better performance than existing smoke recognition methods. For smoke recognition, our PSNet achieves a 96.95% detection rate, 3.02% false alarm rate, and 0.9694 F1-score. The average calculation time for each image is only 0.0195. For smoke component separation, our PSNet also achieves 0.0014 on evaluation criteria mean square error between predicted smoke component images and labelled smoke component images. These key experimental results are better than those of previous methods.

STFE-Net: A Spatial-Temporal Feature Extraction Network for Continuous Sign Language Translation

The main challenge of continuous sign language translation (CSLT) lies in the extraction of both discriminative spatial features and temporal features. In this paper, a spatial-temporal feature extraction network (STFE-Net) is proposed for CSLT, which optimally fuses spatial and temporal features, extracted by the spatial feature extraction network (SFE-Net) and the temporal feature extraction network (TFE-Net), respectively. SFE-Net performs pose estimation for the presenters in sign-language videos. Based on COCO-WholeBody, 133 key points are abbreviated to 53 key points, according to the characteristics of the sign language. High-resolution pose estimation is performed on the hands, along with the whole-body pose estimation, to obtain finer-grained hand features. The spatial features extracted by SFE-Net and the sign language words are then fed to TFE-Net, which is based on Transformer with relative position encoding. In this paper, a dataset for Chinese continuous sign language was created and used for evaluation. STFE-Net achieves Bilingual Evaluation Understudy (BLEU-1, BLEU-2, BLEU-3, BLEU-4) scores of 77.59, 75.62, 74.25, 72.14, respectively. Furthermore, our proposed STFE-Net was also evaluated on two public datasets, RWTH-Phoenix-Weather 2014T and CLS. The BLEU-1, BLEU-2, BLEU-3 and BLEU-4 scores achieved by our method on the former dataset are 48.22, 33.59, 26.41 and 22.45, respectively, and the corresponding scores are 61.54, 58.76, 57.93 and 57.52, respectively, on the latter dataset. Experiment results show that our model can achieve promising performance. If any reader needs the code or dataset, please email lunfee@whut.edu.cn.

Robust Visual Tracking via Multitask Sparse Correlation Filters Learning.

In this article, a novel multitask sparse correlation filters (MTSCF) model, which introduces multitask sparse learning into the CFs framework, is proposed for visual tracking. Specifically, the proposed MTSCF method exploits multitask learning to take the interdependencies among different visual features (e.g., histogram of oriented gradient (HOG), color names, and CNN features) into account to simultaneously learn the CFs and make the learned filters enhance and complement each other to boost the tracking performance. Moreover, it also performs feature selection to dynamically select discriminative spatial features from the target region to distinguish the target object from the background. A l2,1 regularization term is considered to realize multitask sparse learning. In order to solve the objective model, alternating direction method of multipliers is utilized for learning the CFs. By considering multitask sparse learning, the proposed MTSCF model can fully utilize the strength of different visual features and select effective spatial features to better model the appearance of the target object. Extensive experiment results on multiple tracking benchmarks demonstrate that our MTSCF tracker achieves competitive tracking performance in comparison with several state-of-the-art trackers.

MCGRAM: Linking Multi-Scale CNN With a Graph-Based Recurrent Attention Model for Subject-Independent ERP Detection

Electroencephalogram (EEG)-based event-related potential (ERP) detection is expected to play an important role in many applications. However, limited by the low signal-to-noise ratio and large inter-subject variability of EEG signals, it is still challenging to design a high-precision ERP detection method in subject-independent scenarios. In this brief, a novel graph-based multi-scale convolutional recurrent attention model (MCGRAM) is proposed to extract the underlying invariant features of EEG signals for subject-independent ERP detection. Specifically, considering the frequency factors of ERP signals, the multi-scale convolution module is designed for learning the frequency representations. Then, the graph convolution module is developed by constructing the links between each EEG node and designing trainable weights of the links to extract the discriminative spatial features. Besides, two-layer LSTM combined with a self-attention model is utilized to extract temporal representations of different EEG slices. Compared to the existing state-of-the-art methods, MCGRAM achieves the best performance in terms of AUC and F1-Score on a rapid serial visual presentation benchmark dataset. Lastly, the ablation studies demonstrate the superiority of each component in MCGRAM.

A convolution neural network with multi-level convolutional and attention learning for classification of cancer grades and tissue structures in colon histopathological images

A clinically comparable Convolutional Neural Network framework-based technique for performing automated classification of cancer grades and tissue structures in hematoxylin and eosin-stained colon histopathological images is proposed in this paper. It comprised of Enhanced Convolutional Learning Modules (ECLMs), multi-level Attention Learning Module (ALM), and Transitional Modules (TMs). The ECLMs perform a dual mechanism to extract multi-level discriminative spatial features and model cross-channel correlations with fewer computations and effectual avoidance of vanishing gradient issues. The ALM performs focus-refinement through the channel-wise elemental attention learning to accentuate the discriminative channels of the features maps specifically belonging to the important pathological regions and the scale-wise attention learning to facilitate recalibration of features maps at diverse scales. The TMs concatenate the output of these two modules, infuse deep multi-scalar features and eliminate resolution degradation issues. Varied pre-processing techniques are further employed to improvise the generalizability of the proposed network. For performance evaluation, four diverse publicly available datasets (Gland Segmentation challenge(GlaS), Lung Colon(LC)-25000, Kather_Colorectal_Cancer_Texture_Images (Kather-5k), and NCT_HE_CRC_100K(NCT-100k)) and a private dataset Hospital Colon(HosC) are used that further aids in building network invariance against digital variability that exists in real-clinical data. Also, multiple pathologists are involved at every stage of the proposed research and their verification and approval are taken for each step outcome. For the cancer grade classification, the proposed model achieves competitive results for GlaS (Accuracy(97.5%), Precision(97.67%), F1-Score(97.67%), and Recall(97.67%)), LC-25000 (Accuracy(100%), Precision(100%), F1-Score(100%), and Recall(100%)), and HosC(Accuracy(99.45%), Precision(100%), F1-Score(99.65%), and Recall(99.31%)), and while for the tissue structure classification, it achieves results for Kather-5k(Accuracy(98.83%), Precision(98.86%), F1-Score(98.85%), and Recall(98.85%)) and NCT-100k(Accuracy(97.7%), Precision(97.69%), F1-Score(97.71%), and Recall(97.73%)). Furthermore, the reported activation mappings of Gradient-Weighted Class Activation Mappings(Grad-CAM), Occlusion Sensitivity, and Local Interpretable Model-Agnostic Explanations (LIME) evidence that the proposed model can itself learn the similar patterns considered pertinent by the pathologists exclusive of any prerequisite for annotations. In addition, these visualization results are inspected by multiple expert pathologists and provided with a validation score as (GlaS(9.251), LC-25000(9.045), Kather-5k(9.248), NCT-100k(9.262), and HosC(9.853)). This model will provide a secondary referential diagnosis for the pathologists to ease their load and aid them in devising an accurate diagnosis and treatment plan.

Hyperspectral Image Classification via a Novel Spectral–Spatial 3D ConvLSTM-CNN

In recent years, deep learning-based models have produced encouraging results for hyperspectral image (HSI) classification. Specifically, Convolutional Long Short-Term Memory (ConvLSTM) has shown good performance for learning valuable features and modeling long-term dependencies in spectral data. However, it is less effective for learning spatial features, which is an integral part of hyperspectral images. Alternatively, convolutional neural networks (CNNs) can learn spatial features, but they possess limitations in handling long-term dependencies due to the local feature extraction in these networks. Considering these factors, this paper proposes an end-to-end Spectral-Spatial 3D ConvLSTM-CNN based Residual Network (SSCRN), which combines 3D ConvLSTM and 3D CNN for handling both spectral and spatial information, respectively. The contribution of the proposed network is twofold. Firstly, it addresses the long-term dependencies of spectral dimension using 3D ConvLSTM to capture the information related to various ground materials effectively. Secondly, it learns the discriminative spatial features using 3D CNN by employing the concept of the residual blocks to accelerate the training process and alleviate the overfitting. In addition, SSCRN uses batch normalization and dropout to regularize the network for smooth learning. The proposed framework is evaluated on three benchmark datasets widely used by the research community. The results confirm that SSCRN outperforms state-of-the-art methods with an overall accuracy of 99.17%, 99.67%, and 99.31% over Indian Pines, Salinas, and Pavia University datasets, respectively. Moreover, it is worth mentioning that these excellent results were achieved with comparatively fewer epochs, which also confirms the fast learning capabilities of the SSCRN.

CapsCovNet: A Modified Capsule Network to Diagnose COVID-19 From Multimodal Medical Imaging.

Since the end of 2019, novel coronavirus disease (COVID-19) has brought about a plethora of unforeseen changes to the world as we know it. Despite our ceaseless fight against it, COVID-19 has claimed millions of lives, and the death toll exacerbated due to its extremely contagious and fast-spreading nature. To control the spread of this highly contagious disease, a rapid and accurate diagnosis can play a very crucial part. Motivated by this context, a parallelly concatenated convolutional block-based capsule network is proposed in this article as an efficient tool to diagnose the COVID-19 patients from multimodal medical images. Concatenation of deep convolutional blocks of different filter sizes allows us to integrate discriminative spatial features by simultaneously changing the receptive field and enhances the scalability of the model. Moreover, concatenation of capsule layers strengthens the model to learn more complex representation by presenting the information in a fine to coarser manner. The proposed model is evaluated on three benchmark datasets, in which two of them are chest radiograph datasets and the rest is an ultrasound imaging dataset. The architecture that we have proposed through extensive analysis and reasoning achieved outstanding performance in COVID-19 detection task, which signifies the potentiality of the proposed model.

Privacy-preserving Time-series Medical Images Analysis Using a Hybrid Deep Learning Framework

Time-series medical images are an important type of medical data that contain rich temporal and spatial information. As a state-of-the-art, computer-aided diagnosis (CAD) algorithms are usually used on these image sequences to improve analysis accuracy. However, such CAD algorithms are often required to upload medical images to honest-but-curious servers, which introduces severe privacy concerns. To preserve privacy, the existing CAD algorithms support analysis on each encrypted image but not on the whole encrypted image sequences, which leads to the loss of important temporal information among frames. To meet this challenge, a convolutional-LSTM network, named HE-CLSTM, is proposed for analyzing time-series medical images encrypted by a fully homomorphic encryption mechanism. Specifically, several convolutional blocks are constructed to extract discriminative spatial features, and LSTM-based sequence analysis layers (HE-LSTM) are leveraged to encode temporal information from the encrypted image sequences. Moreover, a weighted unit and a sequence voting layer are designed to incorporate both spatial and temporal features with different weights to improve performance while reducing the missed diagnosis rate. The experimental results on two challenging benchmarks (a Cervigram dataset and the BreaKHis public dataset) provide strong evidence that our framework can encode visual representations and sequential dynamics from encrypted medical image sequences; our method achieved AUCs above 0.94 both on the Cervigram and BreaKHis datasets, constituting a significant margin of statistical improvement compared with several competing methods.

Fully Convolutional Networks and a Manifold Graph Embedding-Based Algorithm for PolSAR Image Classification

With the rapid development of artificial intelligence, how to take advantage of deep learning and big data to classify polarimetric synthetic aperture radar (PolSAR) imagery is a hot topic in the field of remote sensing. As a key step for PolSAR image classification, feature extraction technology based on target decomposition is relatively mature, and how to extract discriminative spatial features and integrate these features with polarized information to maximize the classification accuracy is the core issue. In this context, this paper proposes a PolSAR image classification algorithm based on fully convolutional networks (FCNs) and a manifold graph embedding model. First, to describe different types of land objects more comprehensively, various polarized features of PolSAR images are extracted through seven kinds of traditional decomposition methods. Afterwards, drawing on transfer learning, the decomposed features are fed into multiple parallel and pre-trained FCN-8s models to learn deep multi-scale spatial features. Feature maps from the last layer of each FCN model are concatenated to obtain spatial polarization features with high dimensions. Then, a manifold graph embedding model is adopted to seek an effective and compact representation for spatially polarized features in a manifold subspace, simultaneously removing redundant information. Finally, a support vector machine (SVM) is selected as the classifier for pixel-level classification in a manifold subspace. Extensive experiments on three PolSAR datasets demonstrate that the proposed algorithm achieves a superior classification performance.

Hybrid Network with Attention Mechanism for Detection and Location of Myocardial Infarction Based on 12-Lead Electrocardiogram Signals.

The electrocardiogram (ECG) is a non-invasive, inexpensive, and effective tool for myocardial infarction (MI) diagnosis. Conventional detection algorithms require solid domain expertise and rely heavily on handcrafted features. Although previous works have studied deep learning methods for extracting features, these methods still neglect the relationships between different leads and the temporal characteristics of ECG signals. To handle the issues, a novel multi-lead attention (MLA) mechanism integrated with convolutional neural network (CNN) and bidirectional gated recurrent unit (BiGRU) framework (MLA-CNN-BiGRU) is therefore proposed to detect and locate MI via 12-lead ECG records. Specifically, the MLA mechanism automatically measures and assigns the weights to different leads according to their contribution. The two-dimensional CNN module exploits the interrelated characteristics between leads and extracts discriminative spatial features. Moreover, the BiGRU module extracts essential temporal features inside each lead. The spatial and temporal features from these two modules are fused together as global features for classification. In experiments, MI location and detection were performed under both intra-patient scheme and inter-patient scheme to test the robustness of the proposed framework. Experimental results indicate that our intelligent framework achieved satisfactory performance and demonstrated vital clinical significance.

Deep hybrid dilated residual networks for hyperspectral image classification

This study presents a new architecture for deep convolution networks, end-to-end hybrid dilated residual networks wherein 3D cube images are input for hyperspectral image (HSI) classification, and this is termed as 3D-2D SSHDR. The proposed networks could greatly improve the performance of HSI classification as they select the spectral bands adaptively and avoid the limitations of handcrafted selection. Specifically, 3D spectral residual blocks are initially used to learn discriminant features from rich spectral features. Subsequently, extracted 3D images with spectral features are reshaped into 2D feature maps as the input for spatial feature learning, which does not affect the extraction of spatial-spectral features and could reduce the number of parameters. This is followed by using hybrid dilated convolution (HDC) residual blocks to continue learning discriminative spatial features, expanding the receptive field of the convolution kernel without increasing the computational complexity, and avoiding the gridding effect generated by the dilated convolution. Finally, the proposed networks are trained via supervised learning. Besides, this method could accelerate the convergence speed and alleviate the over-fitting problem due to the added batch normalization layers and the design of a multi-residual structure. Experimental results validate that the performance of the proposed network is superior on several HSI benchmark datasets when compared with that of existing methods.

Online continuous multi-stroke Persian/Arabic character recognition by novel spatio-temporal features for digitizer pen devices

Nowadays, digitizer pens have become front end of many digital devices. The increasing use of this technology has necessitated the need for producing pen-based virtual keyboard systems. Despite attempts to create such systems in English, their absence for Persian/Arabic languages is an obvious defect. The goal of this paper is presenting an online continuous Persian/Arabic character recognition method. A character in Persian/Arabic language is made of two types of signs or strokes: main body and delayed strokes (which may be zero or more sign). In this paper, a set of novel and discriminative spatial features are defined for these strokes. These features then are used in a novel algorithm to create a genetic programming-based decision tree called GPDT. The GPDT and spatio-temporal features are utilized by non-deterministic finite automata (NDFA) to recognize group-related strokes and related characters. The reason for using spatio-temporal features is the sameness of the main body of some Persian/Arabic letters (e.g., “ح، خ، ج، چ”). There are also two other issues related to recognizing Persian/Arabic letters: unknown number of delayed stroke segments and the sameness of delayed strokes placement, which are removed by using an NDFA. In fact, after identifying group of main body with the help of GPDT, each recognized stroke makes a move in NDFA to stop in a character state (final state on the end of a path in NDFA). The proposed algorithm recognizes continuous Persian/Arabic letters and digits with a 92.43% accuracy and isolated letters and digits with 97.52% accuracy.

Generative Adversarial Networks and Probabilistic Graph Models for Hyperspectral Image Classification

High spectral dimensionality and the shortage of annotations make hyperspectral image (HSI) classification a challenging problem. Recent studies suggest that convolutional neural networks can learn discriminative spatial features, which play a paramount role in HSI interpretation. However, most of these methods ignore the distinctive spectral-spatial characteristic of hyperspectral data. In addition, a large amount of unlabeled data remains an unexploited gold mine for efficient data use. Therefore, we proposed an integration of generative adversarial networks (GANs) and probabilistic graphical models for HSI classification. Specifically, we used a spectral-spatial generator and a discriminator to identify land cover categories of hyperspectral cubes. Moreover, to take advantage of a large amount of unlabeled data, we adopted a conditional random field to refine the preliminary classification results generated by GANs. Experimental results obtained using two commonly studied datasets demonstrate that the proposed framework achieved encouraging classification accuracy using a small number of data for training.

Detection of Informal Settlements from VHR Images Using Convolutional Neural Networks

Information about the location and extent of informal settlements is necessary to guide decision making and resource allocation for their upgrading. Very high resolution (VHR) satellite images can provide this useful information, however, different urban settlement types are hard to be automatically discriminated and extracted from VHR imagery, because of their abstract semantic class definition. State-of-the-art classification techniques rely on hand-engineering spatial-contextual features to improve the classification results of pixel-based methods. In this paper, we propose to use convolutional neural networks (CNNs) for learning discriminative spatial features, and perform automatic detection of informal settlements. The experimental analysis is carried out on a QuickBird image acquired over Dar es Salaam, Tanzania. The proposed technique is compared against support vector machines (SVMs) using texture features extracted from grey level co-occurrence matrix (GLCM) and local binary patterns (LBP), which result in accuracies of 86.65% and 90.48%, respectively. CNN leads to better classification, resulting in an overall accuracy of 91.71%. A sensitivity analysis shows that deeper networks result in higher accuracies when large training sets are used. The study concludes that training CNN in an end-to-end fashion can automatically learn spatial features from the data that are capable of discriminating complex urban land use classes.