Convolutional Layers Research Articles

Predicting the subcutaneous temperature in cryolipolysis using deep operator networks

Accurate monitoring of subcutaneous temperature is crucial for the safety and efficacy of cryolipolysis. However, existing measurement and simulation methods often require trade-offs between accuracy, depth, and computational efficiency. This study introduces a novel deep learning architecture, ConvD-DeepONet, specifically designed to predict subcutaneous temperature fields with both high accuracy and efficiency. The model effectively captures spatial information and produces multi-dimensional output, owing to the innovative integration of convolutional layers and the decoder network. An average absolute error (MAE) of 0.0038 ℃ and a root mean square error (RMSE) of 0.0083 ℃ are achieved, resulting in over a 50 % reduction compared to the baseline models. Moreover, each prediction is completed in just 5.9 ms, rendering it 120 times faster than traditional finite element method simulations. These results indicate that ConvD-DeepONet is a promising tool for real-time subcutaneous temperature prediction, with the potential to enhance the safety and efficacy of cryolipolysis.

Efficient degradation representation learning network for remote sensing image super-resolution

The advancements in convolutional neural networks have led to significant progress in image super-resolution (SR) techniques. Nevertheless, it is crucial to acknowledge that current SR methods operate under the assumption of bicubic downsampling as a degradation factor in low-resolution (LR) images and train models accordingly. However, this approach does not account for the unknown degradation patterns present in real-world scenes. To address this problem, we propose an efficient degradation representation learning network (EDRLN). Specifically, we adopt a contrast learning approach, which enables the model to distinguish and learn various degradation representations in realistic images to obtain critical degradation information. We also introduce streamlined and efficient pixel attention to strengthen the feature extraction capability of the model. In addition, we optimize our model with mutual affine convolution layers instead of ordinary convolution layers to make it more lightweight while minimizing performance loss. Experimental results on remote sensing and benchmark datasets show that our proposed EDRLN exhibits good performance for different degradation scenarios, while the lightweight version minimizes the performance loss as much as possible. The Code will be available at: https://github.com/Leilei11111/EDRLN.

Bio-mimicking DNA fingerprint profiling for HLS watermarking to counter hardware IP piracy

The multifaceted, multivendor-based global design supply chain induces hardware threats of intellectual property (IP) piracy for modern computing and electronic systems. Current hardware watermarking techniques fall short either in terms of watermark strength (size of covert constraints generated) or number of security layers/variables involved in the security constraints generation process. This paper presents a novel approach for high level synthesis (HLS) watermarking by bio-mimicking DNA fingerprint profiling to counter hardware IP piracy. The proposed approach effectively captures the vital DNA fingerprint profiling phases such as DNA sequencing, DNA fragmentation, fragment replication, DNA ligase, etc. and bio-mimics them to generate a digital watermarking framework. The presented approach has been demonstrated on convolutional layer and JPEG compression-decompression (CODEC) algorithms that are widely used in several medical and machine learning applications. The proposed approach has been thoroughly compared with several state-of-the-art approaches. The proposed approach depicts superior security in the probability of coincidence of up to ~ 104 and tamper tolerance of up to ~ 10368 at 0% overhead as compared to the prior approaches.

Evaluating the Impact of Convolutional Neural Network Layer Depth on the Enhancement of Inertial Navigation System Solutions

Secure navigation is pivotal for several applications including autonomous vehicles, robotics, and aviation. The inertial navigation system estimates position, velocity, and attitude through dead reckoning especially when external references like GPS are unavailable. However, the three accelerometers and three gyroscopes that compose the system are exposed to various types of errors including bias errors, scale factor errors, and noise, which can significantly degrade the accuracy of navigation constituting also a key vulnerability of this system. This work aims to adopt a supervised convolutional neural network (ConvNet) to address this vulnerability inherent in inertial navigation systems. In addition to this, this paper evaluates the impact of the ConvNet layer's depth on the accuracy of these corrections. This evaluation aims to determine the optimal layer configuration maximizing the effectiveness of error correction in INS (Inertial Navigation System) leading to precise navigation solutions.

Optimizing Convolutional Neural Network Architectures

Convolutional neural networks (CNNs) are commonly employed for demanding applications, such as speech recognition, natural language processing, and computer vision. As CNN architectures become more complex, their computational demands grow, leading to substantial energy consumption and complicating their use on devices with limited resources (e.g., edge devices). Furthermore, a new line of research seeking more sustainable approaches to Artificial Intelligence development and research is increasingly drawing attention: Green AI. Motivated by an interest in optimizing Machine Learning models, in this paper, we propose Optimizing Convolutional Neural Network Architectures (OCNNA). It is a novel CNN optimization and construction method based on pruning designed to establish the importance of convolutional layers. The proposal was evaluated through a thorough empirical study including the best known datasets (CIFAR-10, CIFAR-100, and Imagenet) and CNN architectures (VGG-16, ResNet-50, DenseNet-40, and MobileNet), setting accuracy drop and the remaining parameters ratio as objective metrics to compare the performance of OCNNA with the other state-of-the-art approaches. Our method was compared with more than 20 convolutional neural network simplification algorithms, obtaining outstanding results. As a result, OCNNA is a competitive CNN construction method which could ease the deployment of neural networks on the IoT or resource-limited devices.

Multi-area short-term load forecasting based on spatiotemporal graph neural network

Short term power load forecasting can accurately evaluate the overall power load changes and provide accurate reference for power system operation decision-making. To address the limitations of traditional load forecasting methods, which are unable to capture spatial correlations and simultaneously predict load changes in multiple areas, this paper proposes a load forecasting model based on the spatiotemporal attention convolutional mechanism. The proposed model is composed of two key components: a spatiotemporal attention module and a spatiotemporal convolution module. First, the dynamic spatiotemporal correlations between different electricity load areas are captured and analyzed by using the spatiotemporal attention mechanism. Secondly, the spatial pattern and temporal features of the load data sequence are effectively obtained by spatiotemporal convolutional layers. Finally, this paper verifies the accuracy and effectiveness of the proposed method through a real electricity consumption load dataset. The experimental results demonstrate that, in comparison to various benchmark prediction methods, the proposed method can fully explore and utilize the spatiotemporal correlation between different electricity load areas, thereby improving the accuracy of load prediction.

CLASSIFICATION OF ACUTE LYMPHOBLASTIC LEUKEMIA CELLS USING ARTIFICIAL INTELLIGENCE

Due to the morphological similarity between immature lymphoblasts (cancerous cells) to lymphocytes (non-cancerous cells), detecting Acute Lymphoblastic Leukemia poses a significant challenge for pathologists. These cells, which exhibit a similar pattern, can lead to various errors during the diagnosis of the disease. In this study, the cancerous and non-cancerous cells were classified using 3 different artificial intelligence approaches. In the first approach, the classification process was carried out by training Convolutional Neural Networks in 4 different architectures. In the second approach, a hybrid approach was proposed by combining the convolution layer of the CNN model as the feature extractor with the Support Vector Machine, Naive Bayes and Random Forest algorithms as the classifier. The classification processes were carried out by training the proposed second approach. In the third approach, the classification process was performed using transfer learning process and ResNet50 and VGG16 networks. In all experiments, the effects of hyper-parameter and dataset changes on model performance were also examined. The results obtained by these three approaches were compared using the Accuracy, Precision, Recall, F-score, and AUC performance measures. It was determined that the most successful results were obtained with the 1st approach using the Dataset3.

High-throughput systolic array-based accelerator for hybrid transformer-CNN networks

In this era of Transformers enjoying remarkable success, Convolutional Neural Networks (CNNs) remain highly relevant and useful. Indeed, hybrid Transformer-CNN network architectures, which combine the benefits of both approaches, have achieved impressive results. Vision Transformer (ViT) is a significant neural network architecture that features a convolutional layer as its first layer, primarily built on the transformer framework. However, owing to the distinct computation patterns inherent in attention and convolution, existing hardware accelerators for these two models are typically designed separately and lack a unified approach toward accelerating both models efficiently. In this paper, we present a dedicated accelerator on a field-programmable gate array (FPGA) platform. The accelerator, which integrates a configurable three-dimensional systolic array, is specifically designed to accelerate the inferential capabilities of hybrid Transformer-CNN networks. The Convolution and Transformer computations can be mapped to a systolic array by unifying these operations for matrix multiplication. Softmax and LayerNorm which are frequently used in hybrid Transformer-CNN networks were also implemented on FPGA boards. The accelerator achieved high performance with a peak throughput of 722 GOP/s at an average energy efficiency of 53 GOPS/W. Its respective computation latencies were 51.3 ms, 18.1 ms, and 6.8 ms for ViT-Base, ViT-Small, and ViT-Tiny. The accelerator provided a 12× improvement in energy efficiency compared to the CPU, a 2.3× improvement compared to the GPU, and a 1.5× to 2× improvement compared to existing accelerators regarding speed and energy efficiency.

An effective CNN-MHSA method for the fault diagnosis of ZPW-2000A track circuit

The mainstream methods for fault diagnosis of ZPW-2000A track circuits are overly complex in extracting features from sequence data, and their feature extraction capability is relatively limited, thus impacting the performance of fault diagnosis. To address this issue, we propose a fault diagnosis model for track circuits, named as convolutional neural network incorporating multi-head self-attention mechanism (CNN-MHSA). On one hand, the local features of sequence data are locally sensed and extracted by the convolutional layer; on the other hand, the global features of sequence data are captured by establishing long-distance dependency relationships through the multi-head self-attention layer. The dual extraction of local and global features enables accurate diagnosis of faults. Finally, we collect 27 fault modes and 2 normal operation modes in the simulation system and conduct a large number of numerical experiments. The experimental results show that the fault diagnosis accuracy of the CNN-MHSA model in this paper can reach 97.45%, which is an improvement of 0.64%, 0.44%, 0.44%, 0.44%, 0.33%, 0.22%, and 0.11% compared with models such as Decision Tree, Random Forest, CatBoost, long short-term memory (LSTM), convolutional neural network (CNN), and CNN-LSTM. It also has obvious advantages in evaluation metrics such as precision, recall, Macro-F1, and Micro-F1, as well as other evaluation metrics. Therefore, the model significantly improves the comprehensive performance of ZPW-2000A track circuit fault diagnosis and can be used as a more effective fault diagnosis method.

Sustainable utilization of road assets concerning obscured traffic signs recognition

Traffic sign recognition is crucial for sustainable utilization of road assets. However, on real roads, traffic signs are often obscured, making their recognition challenging. Unfortunately, existing research lacks specific analysis of the impact of traffic sign occlusion and methods to improve recognition in such scenarios. This study quantitatively investigates the relationship between the degree of traffic sign occlusion and recognition accuracy. Moreover, a dedicated deep learning model is proposed, utilizing a multi-scale convolutional stacked input layer, to enhance the recognition of obscured traffic signs. Using the Chinese Traffic Sign Recognition Database (TSRD), this study analyzes the recognition performance of four traffic sign categories: indication signs, prohibition signs, speed limit signs, and warning signs, under different occlusion levels. Three widely used deep learning methods, Visual Geometry Group (VGG), Residual Network (ResNet), and Dense Convolutional Network (DenseNet), are compared with the dedicated model. Experimental results demonstrate a significant decrease in recognition accuracy when traffic signs are obscured. Importantly, the proposed dedicated model outperforms the other methods, achieving accuracies of 80.95%, 90.50%, and 97.22% in the scenarios of 50% occlusion, 25% occlusion, and no occlusion, respectively. This study holds implications for sustainable utilization of road assets.

A generalized thermal comfort model using thermographic images and compact convolutional transformers: Towards scalable and adaptive occupant comfort optimization

Thermal comfort models that account for individual thermal sensations are crucial for optimizing environmental control while maintaining occupant comfort. However, the current approach of developing occupant-specific models is not scalable for new occupants whose data were not used for training, making it impractical for multi-occupant spaces and burdensome for management. To address this, we propose a thermal comfort model based on a subject-independent evaluation approach, capable of generalizing to new individuals not included during model training. This model accurately predicts thermal sensations without the need for occupant-specific models, allowing scalability and adaptability to new occupants. Furthermore, this study considers occupants with face masks, which is essential in environments where masks are required for their wellbeing and performance. The model was developed using Compact Convolutional Transformers which combines convolutional layers and transformer layers, allowing the model to capture both fine-grained local features and broader contextual relationships, making it effective for predicting thermal comfort from thermal images. The model was trained using the Leave-One-Subject-Out (LOSO) and Leave-One-Group-Out (LOGO) training approaches. The LOSO approach achieved high accuracies of 98.04 %, 98.93 %, and 98.88 % for dataset A (participants without face masks), dataset B (participants with face masks), and dataset C (a combination of both), respectively. The LOGO approach achieved accuracies of 99.92 %, 99.40 %, and 99.85 % for datasets A, B, and C, respectively, showing a slightly better prediction performance. The approach presented in this study offers a promising solution for designing accurate generalized models for real-time environmental control to meet the thermal comfort needs of occupants.

Machine Learning Models for Probability Classification in Spectrographic EEG Seizures Dataset

The examination of brain signals, namely the Electroencephalogram (EEG) signals, is an approach to possibly detect seizures of the brain. Due to the nature of these signals, deep learning techniques have offered the opportunity to perform automatic or semi-automatic analysis which could support decision and therapeutical approaches. This paper focuses on the possibility of classifying EEG seizure using convolutional layers (namely EfficientNetV2 architectures, i.e., EfficientNetV2S and EfficientNetV2B2), Long Short-Term Memory (LSTM) units, and fine-tuned mechanisms of attention. We use these techniques to untangle the complexity of these signals and accurately predict seizures. The proposed system provided interesting results with an 86.45% accuracy under the Kullback-Leibler Divergence loss of 0.95. Moreover, these results showed that embedding LSTM layers deeply increases the quality of the results since these layers support the analysis of the spatial-temporal dynamics of the EEG signals. On the other hand, it is important to mention that hardware limitations could affect these results and therefore it is important, when setting this architectural system, to fine-tune the data set and balance the performance vs the computational cost of the process.

TSCB-Net: transformer-enhanced semantic segmentation of surface damage of concrete bridges

Concrete bridges are susceptible to surface damage such as water-corrosion, spalling, and rebar-corrosion, which can pose serious safety risks, and to detect this is crucial to ensure their safety. Computer vision-based methods to detect surface damages in concrete bridges have seen great progress. However, mainstream methods can only locate surface damage but cannot obtain detailed information, such as the proportion of damaged pixels. To address these challenges, this paper propose TSCB-Net, a semantic segmentation model for surface damage detection in concrete bridges. TSCB-Net employs both Transformers and an encoder-decoder structure, offering a strong alternative to detect surface damage in concrete bridges. Its encoder uses improved baselines with pyramid vision transformer (PVTV2) to extract semantic information and spatial details of concrete bridge images, layer by layer, and to capture long-term dependence between features. The decoder incorporates a context information recovery (CIR) module in the final upsampling stage, using convolutional layers to realize a larger receptive field. This module effectively models context information for semantic segmentation and facilitates multi-scale surface damage detection of concrete bridges. In experiments, TSCB-Net achieves an overall mIoU of 75.84% and an mF1 of 85.54% when the number of parameters is 5.84 M, outperforming state-of-the-art models. TSCB-Net can quickly and accurately detect and identify surface bridge damage, which can help in the early detection of structural problems and the adoption of appropriate repair and protection measures.

MLSNet: a deep learning model for predicting transcription factor binding sites.

Accurate prediction of transcription factor binding sites (TFBSs) is essential for understanding gene regulation mechanisms and the etiology of diseases. Despite numerous advances in deep learning for predicting TFBSs, their performance can still be enhanced. In this study, we propose MLSNet, a novel deep learning architecture designed specifically to predict TFBSs. MLSNet innovatively integrates multisize convolutional fusion with long short-term memory (LSTM) networks to effectively capture DNA-sparse higher-order sequence features. Further, MLSNet incorporates super token attention and Bi-LSTM to systematically extract and integrate higher-order DNA shape features. Experimental results on 165 ChIP-seq (chromatin immunoprecipitation followed by sequencing) datasets indicate that MLSNet consistently outperforms several state-of-the-art algorithms in the prediction of TFBSs. Specifically, MLSNet reports average metrics: 0.8306 for ACC, 0.8992 for AUROC, and 0.9035 for AUPRC, surpassing the second-best methods by 1.82%, 1.68%, and 1.54%, respectively. This research delineates the effectiveness of combining multi-size convolutional layers with LSTM and DNA shape-based features in enhancing predictive accuracy. Moreover, this study comprehensively assesses the variability in model performance across different cell lines and transcription factors. The source code of MLSNet is available at https://github.com/minghaidea/MLSNet.

Machine Learning-Based Gesture Recognition Glove: Design and Implementation.

In the evolving field of human-computer interaction (HCI), gesture recognition has emerged as a critical focus, with smart gloves equipped with sensors playing one of the most important roles. Despite the significance of dynamic gesture recognition, most research on data gloves has concentrated on static gestures, with only a small percentage addressing dynamic gestures or both. This study explores the development of a low-cost smart glove prototype designed to capture and classify dynamic hand gestures for game control and presents a prototype of data gloves equipped with five flex sensors, five force sensors, and one inertial measurement unit (IMU) sensor. To classify dynamic gestures, we developed a neural network-based classifier, utilizing a convolutional neural network (CNN) with three two-dimensional convolutional layers and rectified linear unit (ReLU) activation where its accuracy was 90%. The developed glove effectively captures dynamic gestures for game control, achieving high classification accuracy, precision, and recall, as evidenced by the confusion matrix and training metrics. Despite limitations in the number of gestures and participants, the solution offers a cost-effective and accurate approach to gesture recognition, with potential applications in VR/AR environments.

Dual-Stream Architecture Enhanced by Soft-Attention Mechanism for Plant Species Classification.

Plants play a vital role in numerous domains, including medicine, agriculture, and environmental balance. Furthermore, they contribute to the production of oxygen and the retention of carbon dioxide, both of which are necessary for living beings. Numerous researchers have conducted thorough research in the classification of plant species where certain studies have focused on limited numbers of classes, while others have employed conventional machine-learning and deep-learning models to classify them. To address these limitations, this paper introduces a novel dual-stream neural architecture embedded with a soft-attention mechanism specifically developed for accurately classifying plant species. The proposed model utilizes residual and inception blocks enhanced with dilated convolutional layers for acquiring both local and global information. Following the extraction of features, both streams are combined, and a soft-attention technique is used to improve the distinct characteristics. The efficacy of the model is shown via extensive experimentation on varied datasets, including several plant species. Moreover, we have contributed a novel dataset that comprises 48 classes of different plant species. The results demonstrate a higher level of performance when compared to current models, emphasizing the capability of the dual-stream design in improving accuracy and model generalization. The integration of a dual-stream architecture, dilated convolutions, and soft attention provides a strong and reliable foundation for the botanical community, supporting advancement in the field of plant species classification.

Decoupling visual and identity features for adversarial palm-vein image attack

Palm-vein has been widely used for biometric recognition due to its resistance to theft and forgery. However, with the emergence of adversarial attacks, most existing palm-vein recognition methods are vulnerable to adversarial image attacks, and to the best of our knowledge, there is still no study specifically focusing on palm-vein image attacks. In this paper, we propose an adversarial palm-vein image attack network that generates highly similar adversarial palm-vein images to the original samples, but with altered palm-identities. Unlike most existing generator-oriented methods that directly learn image features via concatenated convolutional layers, our proposed network first maps palm-vein images into multi-scale high-dimensional shallow representation, and then develops attention-based dual-path feature learning modules to extensively exploit diverse palm-vein-specific features. After that, we design visual-consistency and identity-aware loss functions to specially decouple the visual and identity features to reconstruct the adversarial palm-vein images. By doing this, the visual characteristics of palm-vein images can be largely preserved while the identity information is removed in the adversarial palm-vein images, such that high-aggressive adversarial palm-vein samples can be obtained. Extensive white-box and black-box attack experiments conducted on three widely used databases clearly show the effectiveness of the proposed network.

Prediction of flashover time in a compartment fire by CNN-LSTM based deep neural network considering wearable data collection equipment

Real-time prediction of flashover in a compartment fire is of great significance for rescue. This paper investigated a CNN-LSTM neural network to predict the flashover based on the temperature and radiative heat flux in 24 compartment fire cases simulated with FDS. Multiple conditions affecting compartment fires have been taken into account, like room size, room height, fire source location, opening size, number of vents, and fire load. Radiative heat flux meters and thermocouples were positioned at specific heights, to simulate sensors worn by firefighters entering a fire scene. The proposed model consisted of a convolutional layer and three LSTM layers. After training, the deep learning model effectively identified the flashover in compartment fires, with an average relative error of 4.1 % for temperature prediction and 11.2 % for radiative heat flux prediction. In addition, the model was validated on other simulated fire cases and full-scale experimental data, and the results showed good performance. In this paper, the sensitivity of the model to lead time was analyzed, and the application range of the model was determined. The model performed well within the lead time of 25s. In view of the strong performance of the proposed deep learning model in predicting flashover based on wearable sensors (temperature and radiative heat flux), it is believed that this model holds potential for application in a broader range of fire scenarios. This demonstrates the feasibility of using deep learning artificial intelligence models to predict compartment fire development, providing scientific guidance for smart firefighting technology.

Impact Load Localization Based on Multi-Scale Feature Fusion Convolutional Neural Network.

In order to achieve impact load localization of complex structures such as ships, this paper proposes a multi-scale feature fusion convolutional neural network (MSFF-CNN) method for impact load localization. An end-to-end machine learning model is used, where the raw vibration signals of impact loads are directly fed into the network model to avoid the process of feature extraction. Automatic feature learning and feature concatenation of the signal are achieved through four independent convolutional layers, each using a different size of convolutional kernel. Data normalization and L2 regularization techniques are introduced to enhance the data and prevent overfitting. Classification and localization of impact loads are accomplished using a softmax classification layer. Validation experiments are carried out using a ship's stern compartment model. Our results show that the classification and localization accuracy of the impact load sample group of MSFF-CNN reaches 94.29% compared with a traditional CNN. The method further improves the ability of the network to extract state features, takes local perception and global vision into account, effectively improves the classification ability of the model, and has good prospects for engineering applications.

Specular highlight removal using Quaternion transformer

Specular highlight removal is a very important issue, because specular highlight reflections in images with illumination changes can give very negative effects on various computer vision and image processing tasks. Numerous state-of-the-art networks for the specular removal use convolutional neural networks (CNN), which cannot learn global context effectively. They capture spatial information while overlooking 3D structural correlation information of an RGB image. To address this problem, we introduce a specular highlight removal network based on Quaternion transformer (QformerSHR), which employs a transformer architecture based on Quaternion representation. In particular, a depth-wise separable Quaternion convolutional layer (DSQConv) is proposed to enhance computational performance of QformerSHR, while efficiently preserving the structural correlation of an RGB image by utilizing the Quaternion representation. In addition, a Quaternion transformer block (QTB) based on DSQConv learns global context. As a result, QformerSHR consisting of DSQConv and QTB performs the specular removal from natural and text image datasets effectively. Experimental results demonstrate that it is significantly more effective than state-of-the-art networks for the specular removal, in terms of both quantitative performance and subjective quality.