Training Process Of Convolutional Neural Network Research Articles

An Analysis of Convolutional Neutral Network for Image Classification

Abstract: Convolutional Neural Networks (CNNs) have emerged as a powerful tool for image classification tasks, owing to their ability to automatically learn hierarchical representations from raw pixel data. This paper presents a comprehensive analysis of CNNs for image classification, focusing on their architecture, training process, and performance evaluation metrics. The study investigates various CNN architectures, including AlexNet, GoogLeNet, and ResNet, highlighting their respective strengths and weaknesses in different scenarios. Moreover, the paper explores the impact of hyperparameters such as network depth, filter size, and pooling strategies on classification accuracy and computational efficiency. Furthermore, the training process of CNNs, encompassing data preprocessing, augmentation techniques, and optimization algorithms, is scrutinized to elucidate best practices for achieving optimal performance. Additionally, the paper discusses commonly used evaluation metrics such as accuracy, precision, recall, and F1-score, elucidating their interpretation and significance in assessing model performance. Through a systematic review of existing literature and experimental validation, this analysis aims to provide insights into the underlying mechanisms of CNNs for image classification tasks. Finally, the paper outlines future research directions, emphasizing the need for exploring novel architectures, optimizing hyperparameters, and enhancing interpretability and robustness of CNN models. Overall, this study contributes to a deeper understanding of CNNs for image classification and provides valuable guidance for practitioners and researchers in the field.

A non-intrusive load recognition method combining adaptive PSO algorithm and CNN model

Real-time monitoring of electricity usage details through load monitoring techniques is a crucial aspect of smart power grid management and monitoring, allowing for the acquisition of information on the electricity usage of individual appliances for power users. Accurate detection of electricity load is essential for refined load management and monitoring of power supply quality, facilitating the improvement of power management at the user side and enhancing power operation efficiency. Non-intrusive load monitoring (NILM) techniques require only the analysis of total load data to achieve load monitoring of electricity usage details, and offer advantages such as low cost, easy implementation, high reliability, and user acceptance. However, with the increasing number of distributed new load devices on the user side and the diversification of device development, simple load recognition algorithms are insufficient to meet the identification needs of multiple devices and achieve high recognition accuracy. To address this issue, a non-intrusive load recognition (NILR) model that combines an adaptive particle swarm optimization algorithm (PSO) and convolutional neural network (CNN) has been proposed. In this model, pixelated images of different electrical V-I trajectories are used as inputs for the CNN, and the optimal network layer and convolutional kernel size are determined by the adaptive PSO optimization algorithm during the CNN training process. The proposed model has been validated on the public dataset PLAID, and experimental results demonstrate that it has achieved a overall recognition accuracy of 97.26% and F-1 score of 96.92%, significantly better than other comparison models. The proposed model effectively reduces the confusion between various devices, exhibiting good recognition and generalization capabilities.

Klasifikasi Hama Serangga pada Pertanian Menggunakan Metode Convolutional Neural Network

Insect pest attacks pose a serious threat that can potentially cause significant losses in agricultural production. Therefore, the effective recognition and control of insect pests are crucial for maintaining agricultural productivity and quality of yields. With the advancement of computer technology and artificial intelligence, computer technology can be utilized to automatically recognize images in object recognition, particularly for insect pest classification using the Convolutional Neural Network (CNN) method with the Xception architecture. CNN is one of the types of deep feed-forward artificial neural networks widely used in digital image analysis and can process data in the form of grid patterns. CNN consists of three types of layers: convolutional layer, pooling layer, and fully connected layer. The use of CNN in this research aims to facilitate the classification of insect pests. The CNN process involves stages of training, testing, and validation on insect pests to determine the classification of images of various insect pest species. This research utilizes 1363 image samples with 13 classes of insect pests. The training process of CNN involves several parameters such as batch size, number of epochs, learning rate, and optimizer. The experiment's results indicate that the best accuracy achieved by this model is 93.81% during the training phase and 81.75% during the validation phase. This demonstrates that the model successfully performs insect pest classification using the CNN method.

Passive Thermography Based Bearing Fault Diagnosis Using Transfer Learning With Varying Working Conditions

Bearing is one of the core components of any rotating machine, and its failure is widespread. This reason drives continuous monitoring and detecting bearing faults during machine operation to warn operators and prevent unforeseen damage. This paper proposes an intelligent Passive Thermography (PTG) based fault diagnosis technique for detection of bearing faults using Convolutional Neural Network (CNN) with Transfer Learning (TL) under varying working conditions. The validation of the proposed method has been done on three different datasets; bearing test rig dataset has been taken as a source domain data while Induction Motor (IM), and Machine Fault Simulator (MFS) datasets have been taken as target domain data. The proposed method enables and speeds up the training process of CNN towards accurate adaptation for fault diagnosis approach in the escalated time frame. The experimental results demonstrated that the suggested approach could successfully learn transferable characteristics from the source domain model, which can cope with the issue of limited availability of training data required for the target domain. The classification accuracy on the target domain dataset were varied in the range of 89 -95.4 % in the case of the IM dataset and 96.5-97.5% in the case of the MFS dataset. Moreover, it shows the benefits of the suggested method, which may be utilized as an effective non-invasive diagnostic tool for rotating machines to avoid unexpected system shutdowns.

EMSCNet: Efficient Multisample Contrastive Network for Remote Sensing Image Scene Classification

Significant progress has been achieved in remote sensing image scene classification (RSISC) with the development of convolutional neural networks (CNNs) and vision transformers (ViT). However, high intra-class diversity and inter-class similarity are still enormous challenges for RSISC. Metric learning can effectively improve the discriminative ability of deep representations by constraining the distance between features. Previous metric learning methods only optimize the feature space representation through metric function, ignoring the information interaction between samples. For complex scene images, similarity and discriminative knowledge need to be summarized from the multiple positive and negative pairs. We propose a novel efficient multi-sample contrastive network (EMSCNet) to integrate knowledge from multiple samples. Specifically, we construct a dynamic dictionary with momentum updates to mine positive and negative pairs from the entire dataset. Then, the similarity and discriminative knowledge between samples are summarized by introducing a contrastive module. Finally, the knowledge of the contrastive module is transferred to the backbone classifier through knowledge distillation. The proposed contrastive module can be easily embedded into the training process of CNNs or ViT and removed during inference. Experimental results conducted on three datasets demonstrate the effectiveness of the proposed approach.

Improved generalization performance of convolutional neural networks with LossDA

In recent years, convolutional neural networks (CNNs) have been used in many fields. Nowadays, CNNs have a high learning capability, and this learning capability is accompanied by a more complex model architecture. Complex model architectures allow CNNs to learn more data features, but such a learning process tends to reduce the training model’s ability to generalize to unknown data, and may be associated with problems of overfitting. Although many regularization methods have been proposed, such as data augmentation, batch normalization, and Dropout, research on improving generalization performance is still a common concern in the training process of robust CNNs. In this paper, we propose a dynamically controllable adjustment method, which we call LossDA, that embeds a disturbance variable in the fully-connected layer. The trend of this variable is kept consistent with the training loss, while the magnitude of the variable can be preset to adapt to the training process of different models. Through this dynamic adjustment, the training process of CNNs can be adaptively adjusted. The whole regularization process can improve the generalization performance of CNNs while helping to suppress overfitting. To evaluate this method, this paper conducts comparative experiments on MNIST, FashionMNIST, CIFAR-10, Cats_vs_Dogs, and miniImagenet datasets. The experimental results show that the method can improve the model performance of Light CNNs and Transfer CNNs (InceptionResNet, VGG19, ResNet50, and InceptionV3). The average maximum improvement in accuracy of Light CNNs is 4.62%, F1 is 3.99%, and Recall is 4.69%. The average maximum improvement accuracy of Transfer CNNs is 4.17%, F1 is 5.64%, and Recall is 4.05%.

Acceleration of Deep Neural Network Training Using Field Programmable Gate Arrays.

Convolutional neural network (CNN) training often necessitates a considerable amount of computational resources. In recent years, several studies have proposed for CNN inference and training accelerators in which the FPGAs have previously demonstrated good performance and energy efficiency. To speed up the processing, CNN requires additional computational resources such as memory bandwidth, a FPGA platform resource usage, time, power consumption, and large datasets for training. They are constrained by the requirement for improved hardware acceleration to support scalability beyond existing data and model sizes. This paper proposes a procedure for energy efficient CNN training in collaboration with an FPGA-based accelerator. We employed optimizations such as quantization, which is a common model compression technique, to speed up the CNN training process. Additionally, a gradient accumulation buffer is used to ensure maximum operating efficiency while maintaining gradient descent of the learning algorithm. To validate the design, we implemented the AlexNet and VGG-16 models on an FPGA board and laptop CPU along side GPU. It achieves 203.75 GOPS on Terasic DE1 SoC with the AlexNet model and 196.50 GOPS with the VGG-16 model on Terasic DE-SoC. Our result also exhibits that the FPGA accelerators are more energy efficient than other platforms.

Influence of the parameters of the convolutional neural network model in predicting the effective compressive modulus of porous structure.

In recent years, the convolutional neural network (CNN) technique has emerged as an efficient new method for designing porous structure, but a CNN model generally contains a large number of parameters, each of which could influence the predictive ability of the CNN model. Furthermore, there is no consensus on the setting of each parameter in the CNN model. Therefore, the present study aimed to investigate the sensitivity of the parameters in the CNN model for the prediction of the mechanical property of porous structures. 10,500 samples of porous structure were randomly generated, and their effective compressive moduli obtained from finite element analysis were used as the ground truths to construct and train a CNN model. 8,000 of the samples were used to train the CNN model, 2000 samples were used for the cross-validation of the CNN model and the remaining 500 new structures, which did not participate in the CNN training process, were used to test the predictive power of the CNN model. The sensitivity of the number of convolutional layers, the number of convolution kernels, the number of pooling layers, the number of fully connected layers and the optimizer in the CNN model were then investigated. The results showed that the optimizer has the largest influence on the training speed, while the fully connected layer has the least impact on the training speed. Additionally, the pooling layer has the largest impact on the predictive ability while the optimizer has the least impact on the predictive ability. In conclusion, the parameters of the CNN model play an important role in the performance of the CNN model and the parameter sensitivity analysis can help optimize the CNN model to increase the computational efficiency.

Efficient and portable GEMM-based convolution operators for deep neural network training on multicore processors

Convolutional Neural Networks (CNNs) play a crucial role in many image recognition and classification tasks, recommender systems, brain-computer interfaces, etc. As a consequence, there is a notable interest in developing high performance realizations of the convolution operators, which concentrate a significant portion of the computational cost of this type of neural networks.In a previous work, we introduced a portable, high performance convolution algorithm, based on the BLIS realization of matrix multiplication, which eliminates most of the runtime and memory overheads that impair the performance of the convolution operators appearing in the forward training pass, when performed via explicit im2col transform. In this paper, we extend our ideas to the full training process of CNNs on multicore processors, proposing new high performance strategies to tackle the convolution operators that are present in the more complex backward pass of the training process, while maintaining the portability of the realizations. In addition, we conduct a full integration of these algorithms into a framework for distributed training of CNNs on clusters of computers, providing a complete experimental evaluation of the actual benefits in terms of both performance and memory consumption. Compared with baseline implementation, the use of the new convolution operators using pre-allocated memory can accelerate the training by a factor of about 6%–25%, provided there is sufficient memory available. In comparison, the operator variants that do not rely on persistent memory can save up to 70% of memory.

Heavy-loaded Vehicles Detection Model Testing using Synthetic Dataset

Currently, many roads in Indonesia are damaged. This is due to the presence of large vehicles and large loads that often pass. The more omissions are carried out, the more damaged and severe the road is. The central government and local governments often carry out road repairs, but this problem is often a problem. Damaged roads are indeed many factors, one of which is the road load. The road load is caused by the number of vehicles that carry more than the specified capacity. There are many methods used to monitor roads for road damage. The weighing post is a means used by the government in conducting surveillance. This research is not a proposal to monitor the road, but this is only to create a model for the purpose of detecting heavily or lightly loaded vehicles. This research is to classify using Convolutional Neural Network (CNN) with pre-trained Resnet50. The model generated from the Convolutional Neural Network training process reaches above 90%. Generate Image deep learning algorithms such as the Generative Adversarial Network currently generate a lot of synthetic images. The testing dataset that will be used is generated from style transfer. The model is tested using a testing dataset from the generated style transfer. Style transfer is a method of generating images by combining image content with image styles. The model is pretty good at around 92% for training and 88% for testing, can it detect image style transfer? The Convolutional Neural Network model is said to be good if it is able to recognize the image correctly, considering that the accuracy of the model is very good. One of the reasons why the training model is good but still makes errors during testing, then the image dataset is overfitting

KLASIFIKASI PENYAKIT MATA MENGGUNAKAN CNN

The eye is one of the organs of the human senses, namely the sense of sight. Eyes has an important function in capturing visual information that is used for daily activities. Eye health is important because vision can't be replaced by anything.Previously, a doctor made a diagnosis of an eye disease using retinal fundus images. But it takes expertise and a long time. Therefore, a classification system was made using the Convolutional Neural Network (CNN). The CNN network is used to recognize the visual pattern of image pixels with minimal preprocessing.The variables used during testing are data and batch size for the CNN training process. The data variables consist of 50 images from each class which are reproduced using mirroring with a total of 1,000 images; 50 images from each class reproduced using rotation totaling 2,000 images; and 275 normal images, 55 diabetic images, 250 glaucoma images, 250 cataract images, and 170 hypertension images totaling 1,000 images. Batch size variables used were 25 and 32. After all models were tested, it was concluded that the model trained using 1,600 images and 32 batch size gave the best results, namely loss: 0.1228 and accuracy: 0.9100.

Neural architecture search via standard machine learning methodologies

<abstract><p>In the context of deep learning, the more expensive computational phase is the full training of the learning methodology. Indeed, its effectiveness depends on the choice of proper values for the so-called hyperparameters, namely the parameters that are not trained during the learning process, and such a selection typically requires an extensive numerical investigation with the execution of a significant number of experimental trials. The aim of the paper is to investigate how to choose the hyperparameters related to both the architecture of a Convolutional Neural Network (CNN), such as the number of filters and the kernel size at each convolutional layer, and the optimisation algorithm employed to train the CNN itself, such as the steplength, the mini-batch size and the potential adoption of variance reduction techniques. The main contribution of the paper consists in introducing an automatic Machine Learning technique to set these hyperparameters in such a way that a measure of the CNN performance can be optimised. In particular, given a set of values for the hyperparameters, we propose a low-cost strategy to predict the performance of the corresponding CNN, based on its behavior after only few steps of the training process. To achieve this goal, we generate a dataset whose input samples are provided by a limited number of hyperparameter configurations together with the corresponding CNN measures of performance obtained with only few steps of the CNN training process, while the label of each input sample is the performance corresponding to a complete training of the CNN. Such dataset is used as training set for a Support Vector Machines for Regression and/or Random Forest techniques to predict the performance of the considered learning methodology, given its performance at the initial iterations of its learning process. Furthermore, by a probabilistic exploration of the hyperparameter space, we are able to find, at a quite low cost, the setting of a CNN hyperparameters which provides the optimal performance. The results of an extensive numerical experimentation, carried out on CNNs, together with the use of our performance predictor with NAS-Bench-101, highlight how the proposed methodology for the hyperparameter setting appears very promising.</p></abstract>

Face Expression Recognition using CNN

The emotions evolved in face have an excellent influence on decisions and arguments about various subjects. In psychological theory, emotional states of an individual are often classified into six main categories: surprise, fear, disgust, anger, happiness and sadness. Automatic extraction of those emotions from the face images can help in human computer interaction also as many other applications. Machine learning algorithms and particularly deep neural network can learn complex features and classify the extracted patterns. In this paper, a deep learning¬based framework is used for human emotion recognition. The proposed framework uses the feature extraction then a Convolutional Neural Network (CNN) for classification. The experimental results show that the proposed methodology increases both of the speed training process of CNN and therefore the recognition accuracy.

Epileptic EEG Signal Classification Using Convolutional Neural Network Based on Multi-Segment of EEG Signal

High performance in the epileptic electroencephalogram (EEG) signal classification is an important step in diagnosing epilepsy. Furthermore, this classification is carried out to determine whether the EEG signal from a person's examination results is categorized as an epileptic signal or not (healthy). Several automated techniques have been proposed to assist neurologists in classifying these signals. In general, these techniques have yielded a high average accuracy in classification, but the performance still needs to be improved. Therefore, we propose a convolutional neural network based on multi-segment of EEG signals to classify epileptic EEG signals. This method is built to overcome data limitations in the convolutional neural network training process and add the ensemble combination process. The multi-segment of EEG signal is formed by splitting the signal without overlapping each channel and converting it into the spectrogram image based on the short-time Fourier transform value. The spectrogram image is then used as input for the convolutional neural network in in-depth training and testing. The convolutional neural network model of the training results is used to classify each EEG signal segment on each test channel before entering the ensemble combination stage for the final classification. To evaluate the performance of our proposed method, we used the Bonn EEG dataset. The dataset consists of five EEG records labelled as A, B, C, D, and E. The experiments on several datasets (AB-C, AB-D, AB-E, AB-CD, AB-CDE, and AB-CD-E) which were arranged from the dataset showed that our proposed method (with segment) performs better than without segment. Our proposed method yielded the best average of classification accuracy which is 99.33%, 100%, 100%, 99.5%, 99.8%, and 99.4% for the AB-C, AB-D, AB-E, AB-CD, AB-CDE, and AB-CD-E.By these results, the proposed method can outperform several other methods on the same dataset.

Real-time defect identification of narrow overlap welds and application based on convolutional neural networks

To improve the quality of narrow overlap welds and reduce cost during the high-strength production, it is essential to detect weld defects promptly by identification of the type of defects to provide solution accordingly. This paper proposes an integrated weld defect identification approach combing eddy current detection with 3D laser scanning based on Convolutional Neural Networks (CNN). The detection principle and equipment of the two detection methods are introduced. To fit the training process of CNN, two set of detection signals are preprocessed: a two-dimensional time-frequency diagram for eddy current signals using continuous wavelet transform and for laser images, weld edges are extracted and divided by region using image convolution and combining with integral graph. CNN model VGG16 is trained afterwards with data collected from one local manufacturer in Shanghai. It is discovered that performance of eddy current and laser image identification on different types of weld defects is different, and the accuracy can be increased with the two methods combined. Last, to achieve real-time detection of narrow overlap welding, a two-stage defect recognition model is built which greatly improves the efficiency of weld defect identification without affecting the accuracy of weld defect identification.

Training Sample Selection Strategy Applied to CNN in Magneto-Thermal Coupled Analysis

Deep learning (DL) has attracted more and more attention in computational electromagnetism. Particularly, the convolutional neural network (CNN) is one of the most popular learning models in DL due to its excellent capacity for feature extraction and convergence. The efficiency of CNN mainly depends on how many training samples are needed to effectively converge the network. The sample preparation process often involves a lot of numerical computations, which can be very expensive and time-consuming. In this article, based on the traditional DL network training procedure, two different approaches, namely adding smart training samples and reference samples, are proposed to help the DL network converge. The smart sample selection is based on a greedy algorithm, which can be applied for both training and reference samples. The influences of these two approaches on the CNN training process are investigated by an example of the coupled magneto-thermal computation applied to a transformer. Numerical results show that the two proposed approaches can significantly help the network to converge and improve the efficiency of the DL model.

Convolutional Neural Network for High-Resolution Cloud Motion Prediction from Hemispheric Sky Images

A novel high-resolution method for forecasting cloud motion from all-sky images using deep learning is presented. A convolutional neural network (CNN) was created and trained with more than two years of all-sky images, recorded by a hemispheric sky imager (HSI) at the Institute of Meteorology and Climatology (IMUK) of the Leibniz Universität Hannover, Hannover, Germany. Using the haze indexpostprocessing algorithm, cloud characteristics were found, and the deformation vector of each cloud was performed and used as ground truth. The CNN training process was built to predict cloud motion up to 10 min ahead, in a sequence of HSI images, tracking clouds frame by frame. The first two simulated minutes show a strong similarity between simulated and measured cloud motion, which allows photovoltaic (PV) companies to make accurate horizon time predictions and better marketing decisions for primary and secondary control reserves. This cloud motion algorithm principally targets global irradiance predictions as an application for electrical engineering and in PV output predictions. Comparisons between the results of the predicted region of interest of a cloud by the proposed method and real cloud position show a mean Sørensen–Dice similarity coefficient (SD) of 94 ± 2.6% (mean ± standard deviation) for the first minute, outperforming the persistence model (89 ± 3.8%). As the forecast time window increased the index decreased to 44.4 ± 12.3% for the CNN and 37.8 ± 16.4% for the persistence model for 10 min ahead forecast. In addition, up to 10 min global horizontal irradiance was also derived using a feed-forward artificial neural network technique for each CNN forecasted image. Therefore, the new algorithm presented here increases the SD approximately 15% compared to the reference persistence model.

Statistical Loss and Analysis for Deep Learning in Hyperspectral Image Classification.

Nowadays, deep learning methods, especially the convolutional neural networks (CNNs), have shown impressive performance on extracting abstract and high-level features from the hyperspectral image. However, the general training process of CNNs mainly considers the pixelwise information or the samples' correlation to formulate the penalization while ignores the statistical properties especially the spectral variability of each class in the hyperspectral image. These sample-based penalizations would lead to the uncertainty of the training process due to the imbalanced and limited number of training samples. To overcome this problem, this article characterizes each class from the hyperspectral image as a statistical distribution and further develops a novel statistical loss with the distributions, not directly with samples for deep learning. Based on the Fisher discrimination criterion, the loss penalizes the sample variance of each class distribution to decrease the intraclass variance of the training samples. Moreover, an additional diversity-promoting condition is added to enlarge the interclass variance between different class distributions, and this could better discriminate samples from different classes in the hyperspectral image. Finally, the statistical estimation form of the statistical loss is developed with the training samples through multivariant statistical analysis. Experiments over the real-world hyperspectral images show the effectiveness of the developed statistical loss for deep learning.

FrequentNet : A Novel Interpretable Deep Learning Model for Image Classification

This paper has proposed a new baseline deep learning model of more benefits for image classification. Different from the convolutional neural network(CNN) practice where filters are trained by back propagation to represent different patterns of an image, we are inspired by a method called PCANet to choose filter vectors from basis vectors in frequency domain like Fourier coefficients or wavelets without back propagation. Researchers have demonstrated that those basis in frequency domain can usually provide physical insights, which adds to the interpretability of the model by analyzing the frequencies selected. Besides, the training process will also be more time efficient, mathematically clear and interpretable compared with the black-box training process of CNN.

Identification of Grape Leaf Diseases Using Convolutional Neural Network

The presence of leaf diseases in grapes can reduce the productivity of grapes and result in losses for farmers. Leaf diseases are mainly caused by bacteria, fungi, virus etc. A proper diagnosis of disease in plants is needed in order to take appropriate control measures. This paper aims to assist in the identification and classification of grape leaf diseases Convolutional Neural Network (CNN). CNN is basically an artificial neural network architecture that requires repeated training processes to get good accuracy. CNN consists of 3 stages, namely Data Input, Feature Learning, and Classification. The implementation of CNN in this study uses Keras libraries that use the python programming language. Keras is a framework created to facilitate learning of computers. The CNN training process using 0.0001 learning rate obtained results with an accuracy rate of 91,37%