Encoder Network Research Articles

A New Language-Independent Deep CNN for Scene Text Detection and Style Transfer in Social Media Images.

Due to the adverse effect of quality caused by different social media and arbitrary languages in natural scenes, detecting text from social media images and transferring its style is challenging. This paper presents a novel end-to-end model for text detection and text style transfer in social media images. The key notion of the proposed work is to find dominant information, such as fine details in the degraded images (social media images), and then restore the structure of character information. Therefore, we first introduce a novel idea of extracting gradients from the frequency domain of the input image to reduce the adverse effect of different social media, which outputs text candidate points. The text candidates are further connected into components and used for text detection via a UNet++ like network with an EfficientNet backbone (EffiUNet++). Then, to deal with the style transfer issue, we devise a generative model, which comprises a target encoder and style parameter networks (TESP-Net) to generate the target characters by leveraging the recognition results from the first stage. Specifically, a series of residual mapping and a position attention module are devised to improve the shape and structure of generated characters. The whole model is trained end-to-end so as to optimize the performance. Experiments on our social media dataset, benchmark datasets of natural scene text detection and text style transfer show that the proposed model outperforms the existing text detection and style transfer methods in multilingual and cross-language scenario.

Модуль стиснення зображень на основі нейро- мережевих автокодувальників

A new approach is proposed to data compression in the form of a neural network module based on the structure of autoencoders, which has the most optimal learning time, compression level and obtains sufficiently clear image reconstruction. The main mechanisms for building the structure of encoder and decoder neural networks, which are used as a module, have been developed. The main data for the reconstruction were selected from the open data set Fashion-MNIST, which allows simplified testing of neural network structures, the process of their training and obtaining results. Approaches to image reproduction using neural network layers of convolution and inverse convolution are analyzed. An analysis of the impact on the quality of the resulting image reconstruction of the structure of the output module, which is used to compress the input image, was carried out. Atypical behavior was found during the increase of layers in the structure of the autoencoder, which did not lead to an increase in the quality of image reproduction. The basic necessity of changing the structural parts of the autoencoder and its application in combination with other technologies to obtain a better reproduction result and the elimination of distortions is highlighted.

Cross-Modal Text Steganography Against Synonym Substitution-Based Text Attack

Steganography has received massive attention from the information-hiding community due to its excellent security for covert communication systems. Existing work focuses on improving security on single-modal media while cross-modal media is less explored. However, cross-modal interaction has become a prevalent social manner on current social networks, which arises potential behavioral security issues of single-modal steganography. In this letter, we propose a novel text steganography to explore the practicability of cross-modal steganography. The proposed scheme is composed with image encoder, message encoder, language model, and message extractor networks, where the generated stego texts are semantically consistent with the input reference image. In addition, current generative text steganography schemes are vulnerable to text attack based on synonym substitution since these heuristic algorithms embed information by constructing a mapping between secret messages and candidate tokens. Thus, we design a text attack layer based on synonym substitution to further improve the robustness of generated stego text. Experiments illustrate the superior performance of the proposed cross-modal steganography scheme in terms of security and robustness.

Universal Multimodal Representation for Language Understanding.

Representation learning is the foundation of natural language processing (NLP). This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs or a shared cross-modal embedding space that is pre-trained on out-of-shelf text-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively. The two sequences of representations are further fused by an attention layer for the interaction of the two modalities. In this study, the retrieval process is controllable and flexible. The universal visual representation overcomes the lack of large-scale bilingual sentence-image pairs. Our method can be easily applied to text-only tasks without manually annotated multimodal parallel corpora. We apply the proposed method to a wide range of natural language generation and understanding tasks, including neural machine translation, natural language inference, and semantic similarity. Experimental results show that our method is generally effective for different tasks and languages. Analysis indicates that the visual signals enrich textual representations of content words, provide fine-grained grounding information about the relationship between concepts and events, and potentially conduce to disambiguation.

DERNNet: Dual Encoding Recurrent Neural Network Based Secure Optimal Routing in WSN

A Wireless Sensor Network (WSN) is constructed with numerous sensors over geographical regions. The basic challenge experienced while designing WSN is in increasing the network lifetime and use of low energy. As sensor nodes are resource constrained in nature, novel techniques are essential to improve lifetime of nodes in WSN. Nodes energy is considered as an important resource for sensor node which are battery powered based. In WSN, energy is consumed mainly while data is being transferred among nodes in the network. Several research works are carried out focusing on preserving energy of nodes in the network and made network to live longer. Moreover, this network is threatened by attacks like vampire attack where the network is loaded by fake traffic. Here, Dual Encoding Recurrent Neural network (DERNNet) is proposed for classifying the vampire nodes s node in the network. Moreover, the Grey Wolf Optimization (GWO) algorithm helps for transferring the data by determining best solutions to optimally select the aggregation points; thereby maximizing battery/lifetime of the network nodes. The proposed method is evaluated with three standard approaches namely Knowledge and Intrusion Detection based Secure Atom Search Routing (KIDSASR), Risk-aware Reputation-based Trust (RaRTrust) model and Activation Function-based Trusted Neighbor Selection (AF-TNS) in terms of various parameters. These existing methods may lead to wastage of energy due to vampire attack, which further reduce the lifetime and increase average energy consumed in the network. Hence, the proposed DERNNet method achieves 31.4% of routing overhead, 23% of end-to-end delay, 78.6% of energy efficiency, 94.8% of throughput, 28.2% of average latency, 92.4% of packet delivery ratio, 85.2% of network lifetime, and 94.3% of classification accuracy.

Heterogeneous Graph Contrastive Learning Network for Personalized Micro-Video Recommendation

Personalized micro-video recommendation has attracted a lot of research attention with the growing popularity of micro-video sharing platforms. Many efforts have been made to consider micro-video recommendation as a matching task and shown promising performance, while they only focus on simple features or multi-modal attribute information. Recently, Graph Neural Networks (GNNs) have been employed in many recommendation tasks and achieved impressive success. However, these GNN-based methods may suffer from the following limitations: (1) fail to capture the heterogeneity of nodes in uservideo bipartite graphs; (2) ignore the non-local (global) semantic correlation information remained in heterogeneous graphs. In this paper, we present a novel approach, Heterogeneous Graph Contrastive Learning Network (HGCL), for personalized microvideo recommendation. To consider heterogeneity in user-video bipartite graphs, we first introduce a heterogeneous graph encoder network for a high-quality representation learning of users and micro-videos. Specifically, we design a random surfing model to generate node-type specific homogeneous graphs to preserve the heterogeneity. Then we propose a graph contrastive learning framework to achieve representation learning on each node-type specific homogeneous graph by maximizing the mutual information between local patches of a graph and the global representation of the entire graph. Finally, a type-crossing objective function is proposed to jointly integrate the node embeddings from different node types to facilitate high-quality representation learning. Experimental results on real-world datasets in the micro-video recommendation task validate the performance of our method, compared with state-of-the-art baseline algorithms.

SMAM: Self and Mutual Adaptive Matching for Skeleton-Based Few-Shot Action Recognition.

This paper focuses on skeleton-based few-shot action recognition. Since skeleton is essentially a sparse representation of human action, the feature maps extracted from it, through a standard encoder network in the few-shot condition, may not be sufficiently discriminative for some action sequences that look partially similar to each other. To address this issue, we propose a self and mutual adaptive matching (SMAM) module to convert such feature maps into more discriminative feature vectors. Our method, named as SMAM-Net, first leverages both the temporal information associated with each individual skeleton joint and the spatial relationship among them for feature extraction. Then, the SMAM module adaptively measures the similarity between labeled and query samples and further carries out feature matching within the query set to distinguish similar skeletons of various action categories. Experimental results show that the SMAM-Net outperforms other baselines on the large-scale NTU RGB+D 120 dataset in the tasks of one-shot and five-shot action recognition. We also report our results on smaller datasets including NTU RGB+D 60, SYSU and PKU-MMD to demonstrate that our method is reliable and generalises well on different datasets. Codes and the pretrained SMAM-Net will be made publicly available.

Semantic Segmentation of Gastrointestinal Tract in MRI Scans Using PSPNet Model With ResNet34 Feature Encoding Network

Semantic Segmentation of Gastrointestinal Tract in MRI Scans Using PSPNet Model With ResNet34 Feature Encoding Network

Dual encoder network with transformer-CNN for multi-organ segmentation

Medical image segmentation is a critical step in many imaging applications. Automatic segmentation has gained extensive concern using a convolutional neural network (CNN). However, the traditional CNN-based methods fail to extract global and long-range contextual information due to local convolution operation. Transformer overcomes the limitation of CNN-based models. Inspired by the success of transformers in computer vision (CV), many researchers focus on designing the transformer-based U-shaped method in medical image segmentation. The transformer-based approach cannot effectively capture the fine-grained details. This paper proposes a dual encoder network with transformer-CNN for multi-organ segmentation. The new segmentation framework takes full advantage of CNN and transformer to enhance the segmentation accuracy. The Swin-transformer encoder extracts global information, and the CNN encoder captures local information. We introduce fusion modules to fuse convolutional features and the sequence of features from the transformer. Feature fusion is concatenated through the skip connection to smooth the decision boundary effectively. We extensively evaluate our method on the synapse multi-organ CT dataset and the automated cardiac diagnosis challenge (ACDC) dataset. The results demonstrate that the proposed method achieves Dice similarity coefficient (DSC) metrics of 80.68% and 91.12% on the synapse multi-organ CT and ACDC datasets, respectively. We perform the ablation studies on the ACDC dataset, demonstrating the effectiveness of critical components of our method. Our results match the ground-truth boundary more consistently than the existing models. Our approach gains more accurate results on challenging 2D images for multi-organ segmentation. Compared with the state-of-the-art methods, our proposed method achieves superior performance in multi-organ segmentation tasks. Graphical Abstract The key process in medical image segmentation.

Adaptive fusion network for RGB-D salient object detection

Existing state-of-the-art RGB-D saliency detection models mainly utilize the depth information as complementary cues to enhance the RGB information. However, depth maps can be easily influenced by environment and hence are full of noises. Thus, indiscriminately integrating multi-modality (i.e., RGB and depth) features may induce noise-degraded saliency maps. In this paper, we propose a novel Adaptive Fusion Network (AFNet) to solve this problem. Specifically, we design a triplet encoder network consisting of three subnetworks to process RGB, depth, and fused features, respectively. The three subnetworks are interlinked and form a grid net to facilitate mutual refinement of these multi-modality features. Moreover, we propose a Multi-modality Feature Interaction (MFI) module to exploit complementary cues between depth and RGB modalities and adaptively fuse the multi-modality features. Finally, we design the Cascaded Feature Interweaved Decoder (CFID) to exploit complementary information between multi-level features and refine them iteratively to achieve accurate saliency detection. Experimental results on six commonly used benchmark datasets verify that the proposed AFNet outperforms 20 state-of-the-art counterparts in terms of six widely adopted evaluation metrics. Source code will be publicly available athttps://github.com/clelouch/AFNet upon paper acceptance.

Semantic Segmentation of UAV Images Based on Transformer Framework with Context Information

With the advances in Unmanned Aerial Vehicles (UAVs) technology, aerial images with huge variations in the appearance of objects and complex backgrounds have opened a new direction of work for researchers. The task of semantic segmentation becomes more challenging when capturing inherent features in the global and local context for UAV images. In this paper, we proposed a transformer-based encoder-decoder architecture to address this issue for the precise segmentation of UAV images. The inherent feature representation of the UAV images is exploited in the encoder network using a self-attention-based transformer framework to capture long-range global contextual information. A Token Spatial Information Fusion (TSIF) module is proposed to take advantage of a convolution mechanism that can capture local details. It fuses the local contextual details about the neighboring pixels with the encoder network and makes semantically rich feature representations. We proposed a decoder network that processes the output of the encoder network for the final semantic level prediction of each pixel. We demonstrate the effectiveness of this architecture on UAVid and Urban Drone datasets, where we achieved mIoU of 61.93% and 73.65%, respectively.

Fully automatic tumor segmentation of breast ultrasound images with deep learning.

Breast ultrasound (BUS) imaging is one of the most prevalent approaches for the detection of breast cancers. Tumor segmentation of BUS images can facilitate doctors in localizing tumors and is a necessary step for computer-aided diagnosis systems. While the majority of clinical BUS scans are normal ones without tumors, segmentation approaches such as U-Net often predict mass regions for these images. Such false-positive problem becomes serious if a fully automatic artificial intelligence system is used for routine screening. In this study, we proposed a novel model which is more suitable for routine BUS screening. The model contains a classification branch that determines whether the image is normal or with tumors, and a segmentation branch that outlines tumors. Two branches share the same encoder network. We also built a new dataset that contains 1600 BUS images from 625 patients for training and a testing dataset with 130 images from 120 patients for testing. The dataset is the largest one with pixel-wise masks manually segmented by experienced radiologists. Our code is available at https://github.com/szhangNJU/BUS_segmentation. The area under the receiver operating characteristic curve (AUC) for classifying images into normal/abnormal categories was 0.991. The dice similarity coefficient (DSC) for segmentation of mass regions was 0.898, better than the state-of-the-art models. Testing on an external dataset gave a similar performance, demonstrating a good transferability of our model. Moreover, we simulated the use of the model in actual clinic practice by processing videos recorded during BUS scans; the model gave very low false-positive predictions on normal images without sacrificing sensitivities for images with tumors. Our model achieved better segmentation performance than the state-of-the-art models and showed a good transferability on an external test set. The proposed deep learning architecture holds potential for use in fully automatic BUS health screening.

Efficient Motion Symbol Detection and Multikernel Learning for AER Object Recognition

Address event representation (AER) vision sensors process visual information by monitoring changes in light intensity and generating event streams. In this article, a threshold mechanism based on the statistical results of the peak membrane potentials is proposed to improve the accuracy and efficiency of the motion symbol detection (MSD) method based on the leaky integrate-and-fire (LIF) neuron model and a peak spiking monitoring unit (PSMU). A multikernel learning algorithm based on the tempotron rule, namely, MK-tempotron, is proposed to improve the antinoise performance of the classifier. In MK-tempotron, different kernels are applied to calculate the post-synaptic membrane potentials (PSPs) according to different input spiking patterns, to counteract the impact of noise on the spiking activities, so that the synaptic weights can be changed in the direction of correct firing. We verified the effectiveness of the threshold mechanism and multikernel learning on a variety of data sets. The experimental results show that among the several recent algorithms based on the temporal encoding and spiking neural network classifier, which have advantages in power consumption and network latency, our method achieved the best accuracy on MNIST-DVS (100 ms) and N-CARS, and it also achieved competitive results on N-MNIST, CIFAR10-DVS, Posture-DVS, Poker-DVS, and MNIST-DVS (200 ms/500 ms/2000 ms).

Learning Dynamic Generative Attention for Single Image Super-Resolution

Attention mechanisms have achieved great success for image super-resolution as they can effectively improve the feature representation ability. However, most attention-based methods produce the static attention weights, which are applied identically for all input samples. This popular attention strategy is difficult to automatically adapt the content variations of each individual input, hence hindering further improvements of the magnification performance. To explore towards resolving this challenge, we propose a variational hybrid network with newly dynamic attention mechanisms for image super-resolution tasks. Specifically, we design a multi-scale variational encoder network to transform the curvature map of an input image into the latent space. This is made possible for randomly generated latent variables to reflect the valuable high-frequency information and recalibrate the main network. We utilize these latent variables to further generate controllable attention weights, which modulate not only frequency parameters of convolutional kernels but also spatial characteristics of feature maps for boosting representation power. Moreover, a curvature-domain loss is designed to help the main network to concentrate more on high-frequency geometric structures. Experimental results have revealed that our method can generate more realistic and visually pleasing high-resolution images in comparison to state-of-the-art methods.

PET and MRI image fusion based on a dense convolutional network with dual attention

The fusion techniques of different modalities in medical images, e.g., Positron Emission Tomography (PET) and Magnetic Resonance Imaging (MRI), are increasingly significant in many clinical applications by integrating the complementary information from different medical images. In this paper, we propose a novel fusion model based on a dense convolutional network with dual attention (CSpA-DN) for PET and MRI images. In our framework, an encoder composed of the densely connected neural network is constructed to extract features from source images, and a decoder network is employed to generate the fused image from these features. Simultaneously, a dual-attention module is introduced in the encoder and decoder to further integrate local features along with their global dependencies adaptively. In the dual-attention module, a spatial attention block is leveraged to extract features of each point from encoder network by a weighted sum of feature information at all positions. Meanwhile, the interdependent correlation of all image features is aggregated via a module of channel attention. In addition, we design a specific loss function including image loss, structural loss, gradient loss and perception loss to preserve more structural and detail information and sharpen the edges of targets. Our approach facilitates the fused images to not only preserve abundant functional information from PET images but also retain rich detail structures of MRI images. Experimental results on publicly available datasets illustrate the superiorities of CSpA-DN model compared with state-of-the-art methods according to both qualitative observation and objective assessment.

Multi-layer defense algorithm against deep reinforcement learning-based intruders in smart grids

The Internet of Energy envisions the next generation of smart grids as a highly interconnected network, including advanced metering infrastructures, distributed energy resources, and bidirectional communication systems. The open architecture of IoE-based smart grid results in manifold security concerns, especially the risk of False Data Injection Attacks. The attack may target the technical aspects of a system since fabricating the network's data misleads power scheduling and routing strategies and interrupts the healthy operation of the power system. Additionally, monetary motivation for the intruder sometimes is the main motivation. The conventional cyber defense strategies are unable to detect well-developed False Data Injection Attacks, particularly once the intruder takes advantage of a Deep Reinforcement Learning-based attack development framework that analyzes the dynamic nature of the smart grids. This paper primarily outlines various possible passive attacks using statistical methods. Then, a reinforcement learning-based intruder as an active attack generator is developed, initialized by modeled passive attacks. The attack generator algorithm can simulate the network environment and subsequently creates unclassified attacks. After creating a dynamic attacker, a multilayer defense framework is developed using Snapshot Ensemble Deep Neural Network and an adoptable Deep Auto Encoder network to detect known and unknown threats. Performance evaluations and a real-world simulation prove that the proposed framework can successfully detect both passive and active, where the accuracy and false positive detection rate of the developed framework are 98.82% and 97.42%, respectively.

DMFF-Net: Densely Macroscopic Feature Fusion Network for Fast Magnetic Resonance Image Reconstruction

The task of fast magnetic resonance (MR) image reconstruction is to reconstruct high-quality MR images from undersampled images. Most of the existing methods are based on U-Net, and these methods mainly adopt several simple connections within the network, which we call microscopic design ideas. However, these considerations cannot make full use of the feature information inside the network, which leads to low reconstruction quality. To solve this problem, we rethought the feature utilization method of the encoder and decoder network from a macroscopic point of view and propose a densely macroscopic feature fusion network for fast magnetic resonance image reconstruction. Our network uses three stages to reconstruct high-quality MR images from undersampled images from coarse to fine. We propose an inter-stage feature compensation structure (IFCS) which makes full use of the feature information of different stages and fuses the features of different encoders and decoders. This structure uses a connection method between sub-networks similar to dense form to fuse encoding and decoding features, which is called densely macroscopic feature fusion. A cross network attention block (CNAB) is also proposed to further improve the reconstruction performance. Experiments show that the quality of undersampled MR images is greatly improved, and the detailed information of MR images is enriched to a large extent. Our reconstruction network is lighter than many previous methods, but it achieves better performance. The performance of our method is about 10% higher than that of the original method, and about 3% higher than that of most existing methods. Compared with the nearest optimal algorithms, the performance of our method is improved by about 0.01–0.45%, and our computational complexity is only 1/14 of these algorithms.

Classification of Radar Targets with Micro-Motion Based on RCS Sequences Encoding and Convolutional Neural Network

Radar cross section (RCS) sequences, an easy-to-obtain target feature with small data volume, play a significant role in radar target classification. However, radar target classification based on RCS sequences has the shortcomings of limited information and low recognition accuracy. In order to overcome the shortcomings of RCS-based methods, this paper proposes a spatial micro-motion target classification method based on RCS sequences encoding and convolutional neural network (CNN). First, we establish the micro-motion models of spatial targets, including precession, swing and rolling. Second, we introduce three approaches for encoding RCS sequences as images. These three types of images are Gramian angular field (GAF), Markov transition field (MTF) and recurrence plot (RP). Third, a multi-scale CNN is developed to classify those RCS feature maps. Finally, the experimental results demonstrate that RP is best at reflecting the characteristics of the target among those three encoding methods. Moreover, the proposed network outperforms other existing networks with the highest classification accuracy.

A Multi-Path Encoder Network for GPR Data Inversion to Improve Defect Detection in Reinforced Concrete

Ground penetrating radar (GPR) has been extensively used in the routine inspection of reinforced concrete structures. However, the signatures in GPR images are reflected electromagnetic waves rather than their actual shapes. The interpretation of GPR data is a mandatory but time- and labor-consuming task. Furthermore, the rebars in the near-surface of concrete cause clutter in the GPR images, which hinders the interpretation of GPR data. This work presents a deep learning network to invert GPR B-scan images to permittivity maps of subsurface structures. The proposed network has a multi-path encoder which enables the network to leverage three kinds of GPR data: the original, migrated, and encoder–decoder-processed GPR data. Each type of processing method is designed to serve a different purpose: the original GPR images retain all the waveforms; the migration method intensifies the vertices of the subsurface anomalies; the encoder–decoder network suppresses rebar clutter and enhances the visibility of the defect echoes. The outputs of three processing methods are jointly used to interpret GPR B-scan images. We demonstrated the superiority of the proposed network by comparing it with a network with a single-path encoder. We also validated the proposed network with synthetic and experimental GPR data. The results indicate that the proposed network effectively reconstructs the defects in the reinforced concrete.

A Network for Landslide Detection Using Large-Area Remote Sensing Images with Multiple Spatial Resolutions

Accurately detecting landslides over a large area with complex background objects is a challenging task. Research in the area suffers from three drawbacks in general. First, the models are mostly modified from typical networks, and are not designed specifically for landslide detection. Second, the images used to construct and evaluate models of landslide detection are limited to one spatial resolution, which struggles to meet the requirements of such relevant applications as emergency response. Third, assessments are primarily carried out by using the training data on different parts of the same study area. This makes it difficult to objectively evaluate the transferability of the model, because ground objects in the same area are distributed with similar spectral characteristics. To respond to the challenges above, this study proposes DeenNet, specifically designed for landslide detection. Different from the widely used encoder–decoder networks, DeenNet maintains multi-scale landslide features by decoding the input feature maps to a large scale before encoding a module. The decoding operation is conducted by deconvolution of the input feature maps, while encoding is conducted by convolution. Our model is trained on two earthquake-triggered landslide datasets, constructed using images with different spatial resolutions from different sensor platforms. Two other landslide datasets of different study areas with different spatial resolutions were used to evaluate the trained model. The experimental results demonstrated an at least 6.17% F1-measure improvement by DeenNet compared with three widely used typical encoder–decoder-based networks. The decoder–encoder network structure of DeenNet proves to be effective in maintaining landslide features, regardless of the size of the landslides in different evaluation images. It further validated the capacity of DeenNet in maintaining landslide features, which provides a strong applicability in the context of applications.