Traditional Convolutional Network Research Articles

A Deep Learning-Based Two-Branch Generative Adversarial Network for Image De-Raining

Raindrops can scatter and absorb light, causing images to become blurry or distorted. To improve image quality by reducing the impact of raindrops, this paper proposes a novel generative adversarial network for image de-raining. The network comprises two parts: a generative network and an adversarial network. The generative network performs image de-raining. The adversarial network determines whether the input image is rain-free or de-rained. The generative network comprises two branches: the A branch, which follows a traditional convolutional network structure, and the U branch, which utilizes a U-Net architecture. The A branch includes a multi-scale module for extracting information at different scales and a residual attention module to reduce redundant information interference. The U branch contains an encoder module designed to address the loss of details and local information caused by conventional down-sampling. To improve the performance of the generative network in image de-raining, this paper employs a relative discriminator incorporating a mean squared error loss. This discriminator better measures the differences between rainy and rain-free images while effectively preventing the occurrence of gradient vanishing. Finally, this study performs visual and quantitative comparisons of the proposed method and existing methods on three established rain image datasets. In the quantitative experiments, the proposed method outperforms existing methods regarding PSNR, SSIM, and VIF metrics. Specifically, our method achieves an average PSNR, SSIM, and VIF of approximately 5%, 3%, and 4% higher than the MFAA-GAN method, respectively. These results indicate that the de-rained images generated via the proposed method are closer to rain-free images.

Research on vibration pattern recognition based on phase‐sensitive optical time domain reflectometry and voting fully convolution neural networks

AbstractA method that combines phase‐sensitive optical time domain reflectometry with deep learning to construct new voting fully convolution neural networks (VoteFCNs) is proposed. Compared to the traditional convolutional network, the VoteFCN can be input with data of random size and requires less parameters so that the training speed can be improved greatly. The recognition results can be more accurate and more reliable if we use classification voting count and average recognition rate as the criteria to judge network training quality. At last, the training and identification were conducted by simulating such several disturbance events: walking, raining, climbing fence, hammering the ground optical fibre and normal outdoor environments. The results show that the average test accuracy of this method is about 93.4%.

Research on the state of areas in Ukraine affected by military actions based on remote sensing data and deep learning architectures

The invasion of Ukraine by the Russian Federation and the escalation of military actions in the regions have led to significant damage to residential buildings, civilian infrastructure, various critical infrastructure objects, dams, and extensive pollution of the territories. In this context, the tasks of remote sensing using satellite imagery and aerial observation arise to analyze the impact and conduct an economic assessment of damage in these areas. This work investigates and employs deep neural network (DNNs) models in computer vision (CV) tasks (classification, segmentation) and combines their derivatives, such as convolutional networks (CNNs) and vision transformer models (ViTs), to enhance the accuracy of damage assessment. ViTs have demonstrated significant success, often surpassing traditional CNNs, and have potential applications in remote sensing for damage assessment and the protection of critical infrastructure. The research conducted in this work confirms the importance of applying such technologies in environments where labeled data are rare or non-existent, particularly evaluating the use of DNNs, including CNNs and ViTs, in analyzing regions affected by military actions using synthetic aperture radar (SAR) and multispectral images. The aim and subject of this research also include reviewing the possibilities of combining CNNs and ViTs to improve the speed of image feature extraction, landscape detection, and the detection of complex structural contours of objects, where data are usually insufficient. The results of this study provide a critical review of the application of CNNs and ViTs in remote sensing, identifying significant gaps and challenges, especially in the context of the economic consequences of destruction due to military actions. The technical aspects of using CNNs and transformer-based models for complex CV tasks and transfer learning under data-scarce conditions, as well as the challenges in analyzing large volumes of geophysical data, are considered. The conclusions emphasize the transformational potential of DNNs, especially transformers, in remote sensing under conflict and disaster conditions. Their adaptability and accuracy in various environments underscore their utility in both strategic military and humanitarian contexts, establishing a practical standard for their application in key real, real-world scenario-based territory condition assessment.

LAGSwin: Local attention guided Swin-transformer for thermal infrared sports object detection.

Compared with visible light images, thermal infrared images have poor resolution, low contrast, signal-to-noise ratio, blurred visual effects, and less information. Thermal infrared sports target detection methods relying on traditional convolutional networks capture the rich semantics in high-level features but blur the spatial details. The differences in physical information content and spatial distribution of high and low features are ignored, resulting in a mismatch between the region of interest and the target. To address these issues, we propose a local attention-guided Swin-transformer thermal infrared sports object detection method (LAGSwin) to encode sports objects' spatial transformation and orientation information. On the one hand, Swin-transformer guided by local attention is adopted to enrich the semantic knowledge of low-level features by embedding local focus from high-level features and generating high-quality anchors while increasing the embedding of contextual information. On the other hand, an active rotation filter is employed to encode orientation information, resulting in orientation-sensitive and invariant features to reduce the inconsistency between classification and localization regression. A bidirectional criss-cross fusion strategy is adopted in the feature fusion stage to enable better interaction and embedding features of different resolutions. At last, the evaluation and verification of multiple open-source sports target datasets prove that the proposed LAGSwin detection framework has good robustness and generalization ability.

Improve Unsupervised Machine Learning Model on Fruits by Using VGG-16

The demand for efficient fruit classification has risen with supply chain complexities. For the sake of saving the labor force and improving efficiency, applying machine learning algorithms on fruit categorization to multiple stages in the factory is a feasible solution. The paper briefly delivers the rationale of algorithm VGG-16 and validates its advantage in accuracy by comparing it with other convolutional neural network models. Traditional convolutional network reaches its peak accuracy which is around 40% after 45 training epochs. VGG-16 results with a 97% percent accuracy on the classification of more than 20 kinds of fruits only after 6 epochs and can be improved further by enriching and augmenting the dataset. However, although VGG-16 only requires a small number of epochs, the weighted parameter for each layer increases dramatically and increases the running time. Future researchers should focus on optimizing the algorithm to make it more feasible in the industry. Also, remote computing might be a solution for large computational requirements.

Research on lightweight GPR road surface disease image recognition and data expansion algorithm based on YOLO and GAN

The aim of this paper is to improve the accuracy and efficiency of ground penetrating Radar (GPR) detection of internal road surface disease images. Based on the YOLOv4 target detection algorithm, this study introduces MobileNetV2 and CBAM attention mechanism, and combines the Focal loss confidence loss function to iterate the model, so as to design an efficient and lightweight GPR pavement disease image recognition algorithm, MC-YOLOv4. At the same time, in order to alleviate the problem of data scarcity in GPR, we redesign an unsupervised generative adversarial neural network based on self-attention mechanism, namely SAGAN-W. Experiments show that MC-YOLOv4 not only has faster reasoning ability, but also occupies only 23% of the memory of YOLOv5-S. After using the SAGAN data augmentation, the mAP@ .0.5 evaluation index is further improved by 2.63%, and the collapse and mode collapse that may occur in the training process of the traditional Convolutional Generative Adversarial Neural Network (DCGAN) are avoided. After introducing the Focal loss confidence loss function to participate in the training, It significantly improves the imbalance between the precision and recall of the detection model, and this scheme is verified and supported by real scenes. The experimental results show that the proposed method has significant advantages in automatic detection and data expansion of lightweight GPR pavement invisible diseases, which has a wide range of application value and research significance.

Deep Density Estimation for Cone Counting and Diagnosis of Genetic Eye Diseases From Adaptive Optics Scanning Light Ophthalmoscope Images.

Adaptive optics scanning light ophthalmoscope (AOSLO) imaging offers a microscopic view of the living retina, holding promise for diagnosing and researching eye diseases like retinitis pigmentosa and Stargardt's disease. The technology's clinical impact of AOSLO hinges on early detection through automated analysis tools. We introduce Cone Density Estimation (CoDE) and CoDE for Diagnosis (CoDED). CoDE is a deep density estimation model for cone counting that estimates a density function whose integral is equal to the number of cones. CoDED is an integration of CoDE with deep image classifiers for diagnosis. We use two AOSLO image datasets to train and evaluate the performance of cone density estimation and classification models for retinitis pigmentosa and Stargardt's disease. Bland-Altman plots show that CoDE outperforms state-of-the-art models for cone density estimation. CoDED reported an F1 score of 0.770 ± 0.04 for disease classification, outperforming traditional convolutional networks. CoDE shows promise in classifying the retinitis pigmentosa and Stargardt's disease cases from a single AOSLO image. Our preliminary results suggest the potential role of analyzing patterns in the retinal cellular mosaic to aid in the diagnosis of genetic eye diseases. Our study explores the potential of deep density estimation models to aid in the analysis of AOSLO images. Although the initial results are encouraging, more research is needed to fully realize the potential of such methods in the treatment and study of genetic retinal pathologies.

Ice accretion existence and three-dimensional shape identification based on infrared thermography detection

Ice accretion detection is an important guarantee for production and life safety, especially for aircrafts. In addition, the existence and three-dimensional shape identification of ice accretion are essential to anti/de-icing technology. In this study, an ice accretion detection experiment based on flash infrared thermography system was conducted, followed by recognition approach for identifying the relationship between the infrared signal generates during infrared thermography detection and the shape of ice accretion. Firstly, data dimensionality reduction and mathematical modeling of infrared detection data were performed. Then, a new multi-task model for regression and classification, the multi-task Alexnet-CBAM (Convolutional block attention module) model, was developed through infrared detection of ice accretion samples and other objects. The multi-task Alexnet-CBAM model was exploited for ice thickness prediction as well. Further, the detected ice accretion was three-dimensionally reconstructed after object classification and thickness prediction on the basis of multi-task model regression. The results show that the accuracy of classification and thickness prediction of the multi-task Alexnet-CBAM model surpasses that of traditional convolutional networks. For different shapes of ice accretion, the classification accuracy of the model is higher than 99.91%, and the mean squared error of prediction thickness is less than 0.02. The proposed model is capable of reconstructing three-dimensional shape of ice accretion, which provides a new technology for accurate identification of ice accretion based on infrared thermography detection.

Attention-based deformable convolutional network for Chinese various dynasties character recognition

In this paper, we propose a new deformable convolutional network with attention mechanism to deal with the Chinese character of various dynasties. These ancient Chinese characters are hieroglyph with special spatial structure and gradually evolving into modern Chinese characters. However, because of the intrinsic limitations of traditional convolutional networks in the model, the geometric structure of the convolutional module is fixed. Therefore, we build a new architecture called attention-based deformable networks for this task. Besides, we construct a new dataset called the Chinese characters from Various Dynasties Dataset (CCDD), which includes the evolution of Chinese characters between major dynasties. Deformable convolution based on attention mechanism can obtain those more important offsets, resulting in better performance than the original deformable convolution. In the experimental stage, extensive experiments validate the performance of our approach. Firstly, the effectiveness of this network was verified on two text datasets, Mnist(99.38%) and notMnist(98.22%) both comparing with several modules. Finally, the best performance was achieved on the CCDD dataset(91.50%) compared to other classical deep learning networks. This is the first paper to apply deep learning methods to recognize and classify the characters of major dynasties in China. And archaeological workers need to be able to recognize Chinese characters from various periods in China, and our algorithm can assist them in completing this task to a certain extent. It can also help cultural relic enthusiasts roughly determine the age of cultural relics.

CQ + Training: Minimizing Accuracy Loss in Conversion from Convolutional Neural Networks to Spiking Neural Networks.

Spiking neural networks (SNNs) are attractive for energy-constrained use-cases due to their binarized activation, eliminating the need for weight multiplication. However, its lag in accuracy compared to traditional convolutional network networks (CNNs) has limited its deployment. In this paper, we propose CQ+ training (extended "clamped" and "quantized" training), an SNN-compatible CNN training algorithm that achieves state-of-the-art accuracy for both CIFAR-10 and CIFAR-100 datasets. Using a 7-layer modified VGG model (VGG-*), we achieved 95.06% accuracy on the CIFAR-10 dataset for equivalent SNNs. The accuracy drop from converting the CNN solution to an SNN is only 0.09% when using a time step of 600. To reduce the latency, we propose a parameterized input encoding method and a threshold training method, which further reduces the time window size to 64 while still achieving an accuracy of 94.09%. For the CIFAR-100 dataset, we achieved an accuracy of 77.27% using the same VGG-* structure and a time window of 500. We also demonstrate the transformation of popular CNNs, including ResNet (basic, bottleneck, and shortcut block), MobileNet v1/2, and Densenet, to SNNs with near-zero conversion accuracy loss and a time window size smaller than 60. The framework was developed in PyTorch and is publicly available.

Automatic landslide identification by Dual Graph Convolutional Network and GoogLeNet model-a case study for Xinjiang province, China

Landslides are a natural disaster that exists widely in the world and poses a great threat to human life and property, so it is of great importance to identify and locate landslides. Traditional manual interpretation can effectively identify landslides, but its efficiency is very low for large interpreted areas. In this sense, a landslide recognition method based on the Dual Graph Convolutional Network (DGCNet) is proposed to identify the landslide in remote sensing images quickly and accurately. The remote sensing image (regional remote sensing image) of the northern mountainous area of Tuergen Township, Xinyuan County, Xinjiang Province, was obtained by GeoEye-1 (spatial resolution: 0.5 m). Then, the DGCNet is used to train the labeled images, which finally shows good accuracy of landslide recognition. To show the difference with the traditional convolutional network model, this paper adopts a convolution neural network algorithm named GoogLeNet for image recognition to carry out a comparative analysis, the remote sensing satellite images (single terrain image) of Xinyuan County, Xinjiang Province is used as the data set, and the prediction accuracy is 81.25%. Compared with the GoogLeNet model, the DGCNet model has a larger identification range, which provides a new method for landslide recognition of large-scale regional remote sensing images, but the performance of DGCNet is highly dependent on the quality and characteristics of the input image. If the input data quality is poor or the image structure is unclear, the model’s performance may decline.

X-ray Detection of Prohibited Item Method Based on Dual Attention Mechanism

Prohibited item detection plays a significant role in ensuring public safety, as the timely and accurate identification of prohibited items ensures the safety of lives and property. X-ray transmission imaging technology is commonly employed for prohibited item detection in public spaces, producing X-ray images of luggage to visualize their internal contents. However, challenges such as multiple object overlapping, varying angles, loss of details, and small targets in X-ray transmission imaging pose significant obstacles to prohibited item detection. Therefore, a dual attention mechanism network (DAMN) for X-ray prohibited item detection is proposed. The DAMN consists of three modules, i.e., spatial attention, channel attention, and dependency relationship optimization. A long-range dependency model is achieved by employing a dual attention mechanism with spatial and channel attention, effectively extracting feature information. Meanwhile, the dependency relationship module is integrated to address the shortcomings of traditional convolutional networks in terms of short-range correlations. We conducted experiments comparing the DAMN with several existing algorithms on datasets containing 12 categories of prohibited items, including firearms and knives. The results show that the DAMN has a good performance, particularly in scenarios involving small object detection, detail loss, and target overlap under complex conditions. Specifically, the detection average precision of the DAMN reaches 63.8%, with a segmentation average precision of 54.7%.

Differential convolutional network for noise mask estimation

With the advancements of speech synthesis technology, audio spoof detection systems have become vital for the security of automatic speaker verification systems. Many effective solutions have been offered for clean speech data. However, additive noise has a detrimental impact on the detection performance, as in many other speech related tasks. Noise mask is one of the methods proposed to increase the robustness against the additive noise. The purpose of the noise mask is to identify the time–frequency regions dominated by the noise signal. In this work, differential convolutional neural network is used to create noise masks. Differential convolution considers directional changes of the activations and generates new feature maps. Compared to the traditional convolutional network, a finer noise mask can be created with this method. Once the differential network for the noise masks is trained, its outputs are given to the spoof detection systems. Linear filterbank magnitudes are used as acoustic features for both noise masks and spoof detection. Therefore, the spoof detection systems have 2-channel inputs, i.e., linear filterbank magnitudes and its corresponding mask. Probabilistic linear discriminant analysis (PLDA) with x-vectors, emphasized channel attention, propagation and aggregation time delay neural network (ECAPA–TDNN), and light convolutional neural network (LCNN) followed by long short-term memory layers (LSTM) were used as classifiers. Three different noise types are used in both training and test stages, and two different noise types are used solely in the test stage, to stimulate seen and unseen conditions, respectively. Experiments conducted on the noisy version of ASVspoof 2015 challenge dataset showed that the LCNN–LSTM network with noise masks can achieve superior performance compared to other robust systems and can compete with the state-of-the-art. Considering the average of the known noise types, 2.67% equal error rate (EER) was observed. For the unknown noise types, 3.10% average EER was achieved. For the original (clean) ASVspoof 2015 data, the EER was 0.83%. Additionally, 2.6% EER was observed for logical access condition of ASVspoof 2019 data.

Diagnosis of Chest Pneumonia with X-ray Images Based on Graph Reasoning.

Pneumonia is an acute respiratory infection that affects the lungs. It is the single largest infectious disease that kills children worldwide. According to a 2019 World Health Organization survey, pneumonia caused 740,180 deaths in children under 5 years of age, accounting for 14% of all deaths in children under 5 years of age but 22% of all deaths in children aged 1 to 5 years. This shows that early recognition of pneumonia in children is particularly important. In this study, we propose a pneumonia binary classification model for chest X-ray image recognition based on a deep learning approach. We extract features using a traditional convolutional network framework to obtain features containing rich semantic information. The adjacency matrix is also constructed to represent the degree of relevance of each region in the image. In the final part of the model, we use graph inference to complete the global modeling to help classify pneumonia disease. A total of 6189 children's X-ray films containing 3319 normal cases and 2870 pneumonia cases were used in the experiment. In total, 20% was selected as the test data set, and 11 common models were compared using 4 evaluation metrics, of which the accuracy rate reached 89.1% and the F1-score reached 90%, achieving the optimum.

A CNN-Transformer Network Combining CBAM for Change Detection in High-Resolution Remote Sensing Images

Current deep learning-based change detection approaches mostly produce convincing results by introducing attention mechanisms to traditional convolutional networks. However, given the limitation of the receptive field, convolution-based methods fall short of fully modelling global context and capturing long-range dependencies, thus insufficient in discriminating pseudo changes. Transformers have an efficient global spatio-temporal modelling capability, which is beneficial for the feature representation of changes of interest. However, the lack of detailed information may cause the transformer to locate the boundaries of changed regions inaccurately. Therefore, in this article, a hybrid CNN-transformer architecture named CTCANet, combining the strengths of convolutional networks, transformer, and attention mechanisms, is proposed for high-resolution bi-temporal remote sensing image change detection. To obtain high-level feature representations that reveal changes of interest, CTCANet utilizes tokenizer to embed the features of each image extracted by convolutional network into a sequence of tokens, and the transformer module to model global spatio-temporal context in token space. The optimal bi-temporal information fusion approach is explored here. Subsequently, the reconstructed features carrying deep abstract information are fed to the cascaded decoder to aggregate with features containing shallow fine-grained information, through skip connections. Such an aggregation empowers our model to maintain the completeness of changes and accurately locate small targets. Moreover, the integration of the convolutional block attention module enables the smoothing of semantic gaps between heterogeneous features and the accentuation of relevant changes in both the channel and spatial domains, resulting in more impressive outcomes. The performance of the proposed CTCANet surpasses that of recent certain state-of-the-art methods, as evidenced by experimental results on two publicly accessible datasets, LEVIR-CD and SYSU-CD.

Research on transformer internal defect detection based on large scale model

Aiming at the problem that the traditional convolution network can not be used to detect and identify targets in the more complex target detection application scenarios in power grid business, a target detection scheme based on the ability of large model is proposed. The effects of convolution neural network layers, convolution kernel size and feature dimension on the recognition effect are systematically analyzed and compared. The differences between traditional convolution network and large model in data volume, accuracy and model construction efficiency are compared. The experimental results show that for the specific target detection business scenario downstream of power grid, the model constructed by this scheme has the advantages of fast and efficient, high accuracy and less data dependence, and the comprehensive performance is better than the traditional convolution network detection method.

Spherical Convolution Empowered Viewport Prediction in 360 Video Multicast with Limited FoV Feedback

Field of view (FoV) prediction is critical in 360-degree video multicast, which is a key component of the emerging virtual reality and augmented reality applications. Most of the current prediction methods combining saliency detection and FoV information neither take into account that the distortion of projected 360-degree videos can invalidate the weight sharing of traditional convolutional networks nor do they adequately consider the difficulty of obtaining complete multi-user FoV information, which degrades the prediction performance. This article proposes a spherical convolution-empowered FoV prediction method, which is a multi-source prediction framework combining salient features extracted from 360-degree video with limited FoV feedback information. A spherical convolutional neural network is used instead of a traditional two-dimensional convolutional neural network to eliminate the problem of weight sharing failure caused by video projection distortion. Specifically, salient spatial-temporal features are extracted through a spherical convolution-based saliency detection model, after which the limited feedback FoV information is represented as a time-series model based on a spherical convolution-empowered gated recurrent unit network. Finally, the extracted salient video features are combined to predict future user FoVs. The experimental results show that the performance of the proposed method is better than other prediction methods.

Lightweight and highly robust memristor-based hybrid neural networks for electroencephalogram signal processing

Memristor-based neuromorphic computing shows great potential for high-speed and high-throughput signal processing applications, such as electroencephalogram (EEG) signal processing. Nonetheless, the size of one-transistor one-resistor (1T1R) memristor arrays is limited by the non-ideality of the devices, which prevents the hardware implementation of large and complex networks. In this work, we propose the depthwise separable convolution and bidirectional gate recurrent unit (DSC-BiGRU) network, a lightweight and highly robust hybrid neural network based on 1T1R arrays that enables efficient processing of EEG signals in the temporal, frequency and spatial domains by hybridizing DSC and BiGRU blocks. The network size is reduced and the network robustness is improved while ensuring the network classification accuracy. In the simulation, the measured non-idealities of the 1T1R array are brought into the network through statistical analysis. Compared with traditional convolutional networks, the network parameters are reduced by 95% and the network classification accuracy is improved by 21% at a 95% array yield rate and 5% tolerable error. This work demonstrates that lightweight and highly robust networks based on memristor arrays hold great promise for applications that rely on low consumption and high efficiency.

SRANet: semantic relation aware network for semantic segmentation of remote sensing images

Remote sensing images contain complex feature information, and traditional convolutional networks cannot effectively model the contextual relationships. To address this problem, we propose a semantic segmentation network for remote sensing images based on semantic relationship aware. We construct the semantic relationship aware module to obtain the global semantic information of remote sensing images by self-attention. In addition, the separable space convergence pyramid module was constructed to effectively utilize the feature information in the high-level feature maps. By separable convolution with different dilation rates, the network can acquire multiscale semantic information. Our semantic relation aware network (SRANet) improves the overall accuracy by 0.33% over the benchmark network in the Vaihingen dataset and by 0.42% in the Potsdam dataset. The class activation maps show that the SRANet has ideal activation responses for targets at different scales in images. Furthermore, our SRANet can produce competitive segmentation performance compared with other state-of-the-art segmentation networks.

A Sparse Gating Convolutional Recurrent Network for Traffic Flow Prediction

With the continuous development of deep learning, more and more huge deep learning models are developed by researchers, which leads to an exponential increase of the parameters of models. Therein, the convolutional recurrent network as a type of widely used deep learning method is often employed to handle spatiotemporal data, e.g., traffic data. However, because of the large number of parameters in the model, the convolutional recurrent network needs to consume a lot of computing resources and time in the training process. To reduce the consumption of resources, we propose a sparse convolutional recurrent network with a sparse gating mechanism that is able to reduce the complexity of the network by an improved gate unit while keeping the performance of the model. We evaluate the performance of our proposed network on traffic flow datasets, and the experimental results show that the parameters of the model are significantly reduced under the condition of similar prediction accuracy compared with the traditional convolutional recurrent network.