Stochastic Gradient Descent Optimization Research Articles

Deep learning framework for sensor array precision and accuracy enhancement

In the upcoming years, artificial intelligence is going to transform the practice of medicine in most of its specialties. Deep learning can help achieve better and earlier problem detection, while reducing errors on diagnosis. By feeding a deep neural network (DNN) with the data from a low-cost and low-accuracy sensor array, we demonstrate that it becomes possible to significantly improve the measurements’ precision and accuracy. The data collection is done with an array composed of 32 temperature sensors, including 16 analog and 16 digital sensors. All sensors have accuracies between 0.5 - 2.0;^{^circ } {text{C}}. 800 vectors are extracted, covering a range from to 30 to 45;^circ hbox {C}. In order to improve the temperature readings, we use machine learning to perform a linear regression analysis through a DNN. In an attempt to minimize the model’s complexity in order to eventually run inferences locally, the network with the best results involves only three layers using the hyperbolic tangent activation function and the Adam Stochastic Gradient Descent optimizer. The model is trained with a randomly-selected dataset using 640 vectors (80% of the data) and tested with 160 vectors (20%). Using the mean squared error as a loss function between the data and the model’s prediction, we achieve a loss of only 1.47x10^{-4} on the training set and 1.22x10^{-4} on the test set. As such, we believe this appealing approach offers a new pathway towards significantly better datasets using readily-available ultra low-cost sensors.

Incremental specialized and specialized-generalized matrix factorization models based on adaptive learning rate optimizers

Recommender systems suggest items that are likely to be preferred by a particular user based on historical behavior, actions, and feedback. In real-world applications, data on users and items are continuously generated at a fast pace, such as in e-commerce, social media, digital marketing, and content consumption applications. Since interactions occur over time, these scenarios can be formulated as a data stream where users’ interests are potentially dynamic, i.e., they change over time. Given that changes are expected to occur, one of the current research challenges in streaming recommender systems is that models must adapt their parameters when changes occur to maintain performance. As such changes do not occur for all users and items in the stream at the same time, we consider adapting learning schemes to account for user or item identifiers and model individual parameters. Therefore, we used specialized parameters to adjust the step size for each dataset user or item. More specifically, this study proposes four specialized and specialized-generalized variants of four well-known adaptive learning rate optimizers and shows how they are combined with incremental matrix factorization methods. We tested our proposed optimization strategies on different datasets and showed that one of the proposed specialized variants, that is, InAMSGradUser, improves the RECALL and NDCG rates by up to 11.1 and 7.5 percentage points, respectively, compared to the traditional stochastic gradient descent (SGD) optimizer.

Federated Learning: Crop classification in a smart farm decentralised network

In this paper, the application of federated learning to smart farming has been investigated. The Federated averaging model has been used to carry out crop classification using climatic parameters as independent variables and crop types as labels. The decentralised machine learning models have been used to predict chickpea crops. Through experimentation, it has been observed the model converges when learning rates of 0.001 and 0.01 are considered using the Stochastic gradient descent (SGD) and the Adam optimizers. The model using the Adam optimizer converged faster than the SGD optimizer, this was achieved after 100 epochs. Analysis from the farm dataset has shown that the decentralised models achieve faster convergence and higher accuracy than the centralised network models.

Concurrent multi-peak Bragg coherent x-ray diffraction imaging of 3D nanocrystal lattice displacement via global optimization

In this paper we demonstrated a method to reconstruct vector-valued lattice distortion fields within nanoscale crystals by optimization of a forward model of multi-reflection Bragg coherent diffraction imaging (MR-BCDI) data. The method flexibly accounts for geometric factors that arise when making BCDI measurements, is amenable to efficient inversion with modern optimization toolkits, and allows for globally constraining a single image reconstruction to multiple Bragg peak measurements. This is enabled by a forward model that emulates the multiple Bragg peaks of a MR-BCDI experiment from a single estimate of the 3D crystal sample. We present this forward model, we implement it within the stochastic gradient descent optimization framework, and we demonstrate it with simulated and experimental data of nanocrystals with inhomogeneous internal lattice displacement. We find that utilizing a global optimization approach to MR-BCDI affords a reliable path to convergence of data which is otherwise challenging to reconstruct.

Weight normalization optimization movie recommendation algorithm based on three-way neural interaction networks

Heterogeneous information networks are increasingly used in recommendation algorithms. However, they lack an explicit representation of meta-paths. In using bidirectional neural interaction models for recommendation models, interaction between users and items is often ignored, with an integral impact on the accuracy of the recommendations. To better apply the interaction information, this study proposes a weight-normalized movie recommendation model (SCLW_MCRec) based on a three-way neural interaction network. The model constructs a three-way neural interaction network langle user, meta-path, itemrangle from meta-path contextual information, introducing meta-paths on top of the user-item representation to represent the user-item interaction information. Introduction of a two-layer, one-dimensional convolutional neural network helps capture higher-order interaction features between the user and the item, making the model more powerful in terms of interaction. Adding a dropout layer to the interaction model and using a two-layer convolutional neural network can prevent overfitting and discard irrelevant information features to improve the recommendation. In addition, an extreme cross-entropy loss (argmaxminloss) that incorporates the properties of the argmin and argmax functions is designed to reduce the model loss. A weight-normalization optimization approach is used to better optimize the model and accelerate convergence of the stochastic gradient descent optimization. Compared to current state-of-the-art recommendation models, the SCLW_MCRec model improves the Prec evaluation index by 2.94–35.8%, Recall by 1.15–53.51%, and NDCG by 6.7–49.37% on the MovieLens dataset. The framework provides a significant improvement in recommendation accuracy and also solves the cold-start problem with application of interaction information.

SE_SPnet: Rice leaf disease prediction using stacked parallel convolutional neural network with squeeze‐and‐excitation

AbstractRice is one of the significant crops, and the early identification and prevention of its diseases are essential to ensure adequate and healthy availability to the world's growing population. The use of image processing is an encouraging method for automatic rice leaf disease identification and detection. In particular, the recent advancements indicate the effectiveness of convolutional neural network (CNN) based deep learning approaches. In this direction, the present work proposes a novel stacked parallel convolution layers‐based network (SPnet) with the squeeze‐and‐excitation (SE) architecture, named (SE_SPnet), for classifying diseased rice leaf images. The stacked parallel network block comprises four parallel convolution layers with different kernel sizes for abstractions of the global and local features. The SE block extracts feature information automatically while removing invalid ones. We compare the SE_SPnet model with state‐of‐the‐art CNN models such as VGG16, DenseNet121, and InceptionV3 based on computational effort, accuracy, sensitivity, specificity, precision, recall, and F1‐score. The experimental results show that the SE_SPnet outperforms standard CNN models for the considered rice leaf disease image datasets. In particular, the SE_SPnet achieves the highest accuracy (99.2%), sensitivity (98.2%), specificity (98.5%), precision (98.4%), recall (98.2%), and F1‐score (98.5%) while using stochastic gradient descent (with momentum) optimizer with a 0.01 learning rate. Furthermore, the SE_SPnet also exhibits to outperform when compared with some of the most recent and relevant existing works.

Adaptive learning rate optimization algorithms with dynamic bound based on Barzilai-Borwein method

The training effect of the neural network model is directly influenced by optimization algorithms. The Barzilai-Borwein(BB) method is used in the stochastic gradient descent (SGD) and other deep learning optimization algorithms because of its outstanding performance in terms of convergence speed. In order to improve the stability of BB step size and the output quality of network model in deep learning, this paper presents two improved optimization algorithms based on the BB method: BBbound and AdaBBbound. BBbound reduces the floating BB step size by generating the upper bound of the step in the current iteration. It avoids the occasional occurrence of long steps; AdaBBbound is an adaptive gradient method based on BB method with dynamic bound. The algorithm has a fast convergence effect in the early stage, then calculates learning rate and smoothly transitions to SGD by setting the better conditions of the BB method. We analyzed the performance of the two algorithms based on the initial conditions and the learning rate change curve. Meanwhile, we tested our algorithms on popular network models such as ResNet and DenseNet. The results showed that the new optimization algorithms have achieved high stability and significantly improved the performance of the network model.

Sorting of Fresh Tea Leaf Using Deep Learning and Air Blowing

The sorting of machine-picked fresh tea leaves after mechanized harvesting remains a challenge because of the complex morphological characteristics and physicochemical properties of fresh tea leaves. First, the recognition results of four types of models, namely, YOLOv5, YOLOv3, Fast RCNN, and SSD, were compared. It was found that YOLOv5, with guaranteed recognition accuracy, had a recognition speed of 4.7 ms/frame (about four times that of the second ranked YOLOv3). Therefore, this study presents a novel fresh tea leaf sorting system that provides rapid and high-precision multi-channel sorting for four grades of tea leaves using a tea leaf recognition model based on the You Only Look Once (YOLOv5) deep learning model. Subsequently, a raw dataset, consisting of 6400 target images of different grades and different moisture contents, was used to evaluate three different optimization methods. Among these, the Stochastic Gradient Descent (SGD) optimization method was found to provide the best model training results with an average recognition accuracy of 98.2%. In addition, the recognition efficacy of the recognition model was found to be positively correlated with the gradient coverage of tea’s moisture content in the training set. Theoretical analysis was then conducted, along with the experimental investigation of the air-blowing force on the fresh tea leaves in the sorting process, with 30° determined to be the optimal air-blowing angle. Finally, the overall results showed that the construction of the full moisture content training set enabled a model recognition accuracy of up to 88.8%, a recall of 88.4%, a recognition speed of 4.7 ms/frame, and an overall sorting accuracy of 85.4%. This result is promising for multi-channel sorting of fresh tea leaf grades in complex situations, and as such provides a strong basis for the application of tea leaf sorting equipment.

Comparison Of SGD, Rmsprop, and Adam Optimation In Animal Classification Using CNNs

Many measures have been taken to protect endangered species by using "camera trap" technology which is widespread in the field of technology-based nature protection field research. In this study, a machine learning-based approach is presented to identify endangered wildlife images with a data set containing 5000 images taken from Kaggle and some other sources. The Gradient Descent optimization method is often used for Artificial Neural Network (ANN) training. This method plays a role in finding the weight values that give the best output value. Three optimization methods have been implemented, namely Stochastic Gradient Descent (SGD), ADADELTA, and Adam on the Artificial Neural Network system for animal data classification. In some of the studies reviewed there are differences in the results of SGD and ADAM, which on the one hand SGD is superior, and on the one hand ADAM is superior with the appropriate learning rate. The results of this study show that the CNN method with the Adam optimization function produces the highest accuracy compared to the SGD and RMSprop optimization methods. The model trained using Adam's optimization function achieved an accuracy of 89.81% on the test, showing the feasibility of the approach.

Reducing crosstalk of a multi-plane holographic display by the time-multiplexing stochastic gradient descent.

Multi-plane reconstruction is essential for realizing a holographic three-dimensional (3D) display. One fundamental issue in conventional multi-plane Gerchberg-Saxton (GS) algorithm is the inter-plane crosstalk, mainly caused by the neglect of other planes' interference in the process of amplitude replacement at each object plane. In this paper, we proposed the time-multiplexing stochastic gradient descent (TM-SGD) optimization algorithm to reduce the multi-plane reconstruction crosstalk. First, the global optimization feature of stochastic gradient descent (SGD) was utilized to reduce the inter-plane crosstalk. However, the crosstalk optimization effect would degrade as the number of object planes increases, due to the imbalance between input and output information. Thus, we further introduced the time-multiplexing strategy into both the iteration and reconstruction process of multi-plane SGD to increase input information. In TM-SGD, multiple sub-holograms are obtained through multi-loop iteration and then sequentially refreshed on spatial light modulator (SLM). The optimization condition between the holograms and the object planes converts from one-to-many to many-to-many, improving the optimization of inter-plane crosstalk. During the persistence of vision, multiple sub-hologram jointly reconstruct the crosstalk-free multi-plane images. Through simulation and experiment, we confirmed that TM-SGD could effectively reduce the inter-plane crosstalk and improve image quality.The proposed TM-SGD-based holographic display has wide applications in tomographic 3D visualization for biology, medical science, and engineering design, which need to reconstruct multiple independent tomographic images without inter-plane crosstalk.

An Improvised Sentiment Analysis Model on Twitter Data Using Stochastic Gradient Descent (SGD) Optimization Algorithm in Stochastic Gate Neural Network (SGNN).

Sentiment analysis is one of the effective techniques for mining the opinion from shapeless data contains text like review of the products, review of the movie. Sentiment analysis is used as a key to gather response from consumers, reviews of brands, marketing analyses, and political campaigns. In the subject of natural processing, performing sentiment analysis using the data obtained from Twitter is considered as a new study in these days. The dataset is gathered using the Twitter API and the Twitter package. The analysis of Twitter data is a process which takes place automatically by text data analysis to determine the view of public on the specified topic. Here, an improvised sentimental analysis model is proposed to identify the polarity of the tweets such as positive, neutral and negative. In this paper, stochastic gradient descent (SGD) algorithm uses stochastic gradient neural network (SGNN) to categorize the sentiment analysis on basis of tweets provided by the Twitter users and the proposed stochastic gradient descent optimization Algorithm based on stochastic gradient neural network (SGDOA-SGNN) provides better performance when compared with the existing Forest-Whale Optimization Algorithm based on deep neural network F-WOA-DNN model.

Autonomous landing scene recognition based on transfer learning for drones

In this paper, we study autonomous landing scene recognition with knowledge transfer for drones. Considering the difficulties in aerial remote sensing, especially that some scenes are extremely similar, or the same scene has different representations in different altitudes, we employ a deep convolutional neural network (CNN) based on knowledge transfer and fine-tuning to solve the problem. Then, LandingScenes-7 dataset is established and divided into seven classes. Moreover, there is still a novelty detection problem in the classifier, and we address this by excluding other landing scenes using the approach of thresholding in the prediction stage. We employ the transfer learning method based on ResNeXt-50 backbone with the adaptive momentum (ADAM) optimization algorithm. We also compare ResNet-50 backbone and the momentum stochastic gradient descent (SGD) optimizer. Experiment results show that ResNeXt-50 based on the ADAM optimization algorithm has better performance. With a pre-trained model and fine-tuning, it can achieve 97.845 0% top-1 accuracy on the LandingScenes-7 dataset, paving the way for drones to autonomously learn landing scenes.

Mask usage recognition using vision transformer with transfer learning and data augmentation

The COVID-19 pandemic has disrupted various levels of society. The use of masks is essential in preventing the spread of COVID-19 by identifying an image of a person using a mask. Although only 23.1% of people use masks correctly, Artificial Neural Networks (ANN) can help classify the use of good masks to help slow the spread of the Covid-19 virus. However, it requires a large dataset to train an ANN that can classify the use of masks correctly. MaskedFace-Net is a suitable dataset consisting of 137016 digital images with 4 class labels, namely Mask, Mask Chin, Mask Mouth Chin, and Mask Nose Mouth. Mask classification training utilizes Vision Transformers (ViT) architecture with transfer learning method using pre-trained weights on ImageNet-21k, with random augmentation. In addition, the hyper-parameters of training of 20 epochs, an Stochastic Gradient Descent (SGD) optimizer with a learning rate of 0.03, a batch size of 64, a Gaussian Cumulative Distribution (GeLU) activation function, and a Cross-Entropy loss function are used to be applied on the training of three architectures of ViT, namely Base-16, Large-16, and Huge-14. Furthermore, comparisons of with and without augmentation and transfer learning are conducted. This study found that the best classification is transfer learning and augmentation using ViT Huge-14. Using this method on MaskedFace-Net dataset, the research reaches an accuracy of 0.9601 on training data, 0.9412 on validation data, and 0.9534 on test data. This research shows that training the ViT model with data augmentation and transfer learning improves classification of the mask usage, even better than convolutional-based Residual Network (ResNet).

Computer-Aided Early Melanoma Brain-Tumor Detection Using Deep-Learning Approach.

Brain tumors affect the normal functioning of the brain and if not treated in time these cancerous cells may affect the other tissues, blood vessels, and nerves surrounding these cells. Today, a large population worldwide is affected by the precarious disease of the brain tumor. Healthy tissues of the brain are suspected to be damaged because of tumors that become the most significant reason for a large number of deaths nowadays. Therefore, their early detection is necessary to prevent patients from unfortunate mishaps resulting in loss of lives. The manual detection of brain tumors is a challenging task due to discrepancies in appearance in terms of shape, size, nucleus, etc. As a result, an automatic system is required for the early detection of brain tumors. In this paper, the detection of tumors in brain cells is carried out using a deep convolutional neural network with stochastic gradient descent (SGD) optimization algorithm. The multi-classification of brain tumors is performed using the ResNet-50 model and evaluated on the public Kaggle brain-tumor dataset. The method achieved 99.82% and 99.5% training and testing accuracy, respectively. The experimental result indicates that the proposed model outperformed baseline methods, and provides a compelling reason to be applied to other diseases.

Deep Convolutional Neural Network with a Stochastic Gradient Descent Optimizer (PDCNN-SGD) model for telugu character recognition

Telugu Character Recognition (TCR) has received significant attention because of the drastic increase in technological advancements such as multimedia, smartphones and iPods, and paper documents. Offline character recognition is the process of identifying Telugu characters from the scanned image or document whereas online character recognition enables to recognition of characters by the machine while the user writes. Several researchers have attempted to design online TCR models by the use of distinct classification models and feature extraction approaches. It is still necessary to construct automated and intelligent online TCR models, even if many studies have focused on offline TCR models. The Telugu character dataset construction and validation using an Inception and ResNet-based model are presented. The collection of 645 letters in the dataset includes 18 Achus, 38 Hallus, 35 Othulu, 34*16 Guninthamulu and 10 Ankelu. The proposed technique aims to efficiently recognize and identify distinctive Telugu characters online. This model's main preprocessing steps to achieve its goals include normalization, smoothing, and interpolation. Improved recognition performance can be attained by using Stochastic Gradient Descent (SGD) to optimize the model's hyperparameters.

Strategy for Rapid Diabetic Retinopathy Exposure Based on Enhanced Feature Extraction Processing

In the modern world, one of the most severe eye infections brought on by diabetes is known as diabetic retinopathy, which will result in retinal damage, and, thus, lead to blindness. Diabetic retinopathy can be well treated with early diagnosis. Retinal fundus images of humans are used to screen for lesions in the retina. However, detecting DR in the early stages is challenging due to the minimal symptoms. Furthermore, the occurrence of diseases linked to vascular anomalies brought on by DR aids in diagnosing the condition. Nevertheless, the resources required for manually identifying the lesions are high. Similarly, training for Convolutional Neural Networks is more time-consuming. This proposed research aims to improve diabetic retinopathy diagnosis by developing an enhanced deep learning model for timely DR identification that is potentially more accurate than existing CNN-based models. The proposed model will detect various lesions from retinal images in the early stages. First, characteristics are retrieved from the retinal fundus picture and put into the EDLM for classification. For dimensionality reduction, EDLM is used. Additionally, the classification and feature extraction processes are optimized using the stochastic gradient descent optimizer. The EDLM effectiveness is assessed on the KAG GLE dataset with 3459 retinal images, and results are compared over VGG16, VGG19, RESNET18, RESNET34, and RESNET50.

Eye Detection-Based Deep Belief Neural Networks and Speeded-Up Robust Feature Algorithm

The ability to detect and localize the human eye is critical for use in security applications and human identification and verification systems. This is because eye recognition algorithms have multiple challenges, such as multi-pose variations, ocular parts, and illumination. Moreover, the modern security applications fail to detect facial expressions from eye images. In this paper, a Speeded-Up Roust Feature (SURF) Algorithm was utilized to localize the face images of the enrolled subjects. We highlighted on eye and pupil parts to be detected based on SURF, Hough Circle Transform (HCT), and Local Binary Pattern (LBP). Afterward, Deep Belief Neural Networks (DBNN) were used to classify the input features results from the SURF algorithm. We further determined the correctly and wrongly classified subjects using a confusion matrix with two class labels to classify people whose eye images are correctly detected. We apply Stochastic Gradient Descent (SGD) optimizer to address the overfitting problem, and the hyper-parameters are fine-tuned based on the applied DBNN. The accuracy of the proposed system is determined based on SURF, LBP, and DBNN classifier achieved 95.54% for the ORL dataset, 94.07% for the BioID, and 96.20% for the CASIA-V5 dataset. The proposed approach is more reliable and more advanced when compared with state-of-the-art algorithms.

Multi-label classification of chest X-ray images with pre-trained vision Transformer model

目的 基于计算机的胸腔X线影像疾病检测和分类目前存在误诊率高，准确率低的问题。本文在视觉Transformer（vision Transformer，ViT）预训练模型的基础上，通过迁移学习方法，实现胸腔X线影像辅助诊断，提高诊断准确率和效率。方法 选用带有卷积神经网络（convolutional neural network，CNN）的ViT模型，其在超大规模自然图像数据集中进行了预训练；通过微调模型结构，使用预训练的ViT模型参数初始化主干网络，并迁移至胸腔X线影像数据集中再次训练，实现疾病多标签分类。结果 在IU X-Ray数据集中对ViT迁移学习前、后模型平均AUC（area under ROC curve）得分进行对比分析实验。结果表明，预训练ViT模型平均AUC得分为0.774，与不使用迁移学习相比提升了0.208。并针对模型结构和数据预处理进行了消融实验，对ViT中的注意力机制进行可视化，进一步验证了模型有效性。最后使用Chest X-Ray14和CheXpert数据集训练微调后的ViT模型，平均AUC得分为0.839和0.806，与对比方法相比分别有0.014～0.031的提升。结论 与其他方法相比，ViT模型胸腔X线影像的多标签分类精确度更高，且迁移学习可以在降低训练成本的同时提升ViT模型的分类性能和泛化性。消融实验与模型可视化表明，包含CNN结构的ViT模型能重点关注有意义的区域，高效获取胸腔X线影像的视觉特征。;Objective The chest X-ray-relevant screening and diagnostic method is essential for radiology nowadays. Most of chest X-ray images interpretation is still restricted by clinical experience and challenged for misdiagnose and missed diagnoses. To detect and identify one or more potential diseases in images automatically，it is beneficial for improving diagnostic efficiency and accuracy using computer-based technique. Compared to natural images，multiple lesions are challenged to be detected and distinguished accurately in a single image because abnormal areas have a small proportion and complex representations in chest X-ray images. Current convolutional neural network（CNN）based deep learning models have been widely used in the context of medical imaging. The structure of the CNN convolution kernel has sensitive to local detail information，and it is possible to extract richer image features. However，the convolution kernel cannot be used to get global information，and the features-extracted are restricted of redundant information like its relevance of background， muscles，and bones. The model’s performance in multi-label classification tasks are affected to a certain extent. At present，the vision Transformer（ViT）model has achieved its priorities in computer vision-related tasks. The ViT can be used to capture information simultaneously and effectively for multiple regions of the entire image. However，it is required to use large-scale dataset training to achieve good performance. Due to some factors like patient privacy and manual annotate costs，the size of the chest X-ray image data set has been limited. To reduce the model's dependence on data scale and improve the performance of multi-label classification，we develop the CNN-based ViT pre-training model in terms of the transfer learning method for diagnosis-assisted of chest X-ray image and multi-label classification. Method The CNN-based ViT model is pre-trained on a huge scale ground truth dataset，and it is used to obtain the initial parameters of the model. The model structure is fine-tuned according to the features of chest X-ray dataset. A 1×1 convolution layer is used to convert the chest X-ray images channels between 1 to 3. The number of output nodes of the linear layer in the classifier is balanced from 1 000 to the number of chest X-ray classification labels，and the Sigmoid is used as an activation function. The parameters of the backbone network are initialized in terms of the pre-trained ViT model parameters，and it is trained in the chest X-ray dataset after that to complete multi-label classification. The experiment is configured of Python3. 7 and PyTorch1. 8 to construct the model and RTX3090 GPU for training. Stochastic gradient descent（SGD）optimizer，binary cross-entropy（BCE）loss function，an initial learning rate of 1E-3，the cosine annealing learning rate decay are used. For training，each image is scaled to a size of 512×512 pixels，and a 224×224 pixels area and it is then cropped in random as the model input，and data augmentation is performed randomly by some of the flipping，perspective transformation， shearing，translation，zooming，and changing brightness. For testing，the chest X-ray image is scaled to 256×256 pixels and center crop a 224×224 area to input the trained model. Result The experiment is performed on the IU X-Ray，which is a small-scale chest X-ray dataset. This model is evaluated in quantitative using the average of area under ROC curve （AUC）scores across all classification labels. The results show that the average AUC score of the pre-trained ViT model is 0. 774. The accuracy and training efficiency of the non-pre-trained ViT model is dropped significantly. The average AUC score is reached to 0. 566 only，which is 0. 208 lower. In addition，the attention mechanism heat map is generated based on the ViT model，which can strengthen the interpretability of the model. A series of ablation experiments are carried out for data augmentation，model structure，and batch size design. The fine-tuned ViT model is trained on the Chest-Ray14 and CheXpert dataset as well. The average AUC score is reached to 0. 839 and 0. 806，which is optimized by 0. 014 and 0. 031. Conclusion A pre-trained ViT model is used for the multi-label classification of chest X-ray images via transfer learning. The experimental results illustrate that the ViT has its stronger multi-label classification performance in chest Xray images，and its attention mechanism is beneficial for lesions precision-focused like the interior of the chest cavity and the heart. Transfer learning is potential to improve the classification performance and model generalization of the ViT in small-scale datasets，and the training cost is reduced greatly. Ablation experiments demonstrate that the incorporated model of CNN and Transformer has its priority beyond single-structure model. Data enhancement and the batch size cutting can improve the performance of the model，but smaller scale of batch is still interlinked to longer training span. To improve the model's ability，we predict that future research direction can be focused on the extraction for complex disease and highlevel semantic information，such as their small lesions，disease location，and severity.

DermICNet: Efficient Dermoscopic Image Classification Network for Automated Skin Cancer Diagnosis

The incidence of skin cancer is rapidly increasing worldwide. The relevance of Skin Cancer Diagnosis (SCD) and the difficulty in achieving an accurate and consistent diagnosis have resulted in significant research interest. Furthermore, automated detection or classification would be even more helpful in a diagnostic assistance system. This study develops an efficient Dermoscopic Image Classification Network (DermICNet) for automated SCD. The proposed DermICNet is a deep learning architecture with an efficient arrangement of eight convolutional layers with small-sized convolution filters (3x3). The extracted features from the convolution layers are fed to the dense layer for classification. It consists of a neural network that uses stochastic gradient descent optimization to find the optimal solution for SCD. Finally, a softmax classifier is employed to classify the patterns in the dermoscopic images. The proposed DermICNet is assessed using PH2 database images. The classification results reported are based on the random-split (70:30) approach, which divides the PH2 database into training and testing. It is demonstrated that it is feasible to discriminate between abnormal and normal dermoscopic images with an average accuracy of 99.2% using the proposed DermICNet. The results suggest that the analysis of dermoscopic images using DermICNet has the potential as a diagnostic tool for SCD.

Implementation of CNN and ANN for Fashion-MNIST-Dataset using Different Optimizers

Objectives: The paper presents application of convolution neural network and artificial neural network for image classification problem for clothing dataset along with their performance comparison against different optimizers. The major objective of this paper is to perform image classification on fashion-mnist clothing dataset images. Methods: The methods used here are, the traditional ANN and CNN. Here image classification is performed on Fashion-mnist, clothing dataset using CNN and ANN with different optimizers. The performance of the working of ANN and CNN in classifying images from fashion-mnist dataset is compared against different optimizers namely stochastic gradient Descent, Adagrad, RMS prop and Adam optimizer. Findings: The study found that CNN worked better than ANN yielding training accuracy of 95%, 93% and testing accuracy of 91%, 89% when used with Adam and RmsProp respectively. Novelty: The novelty of this work is to present a comparative study of image classification using CNN, ANN using different optimizers, since not many studies or research articles showed the performance comparison of traditional and convolution neural networks in image classification along with different optimizers. Since the real-world scenarios of today require enormous data to be processed, CNN can fit well to diversify applications since they highly reduce the number of parameters to be trained that speeds up the training process. Moreover, to be specific on image classification problems they require the best and most prominent features to be detected and uncovered; this can be achieved using CNN since it has the concept of convolution using filters at its Core. Hence, CNN is highly recommended for such image classification applications than the traditional artificial-neural-networks because of the aforementioned reasons. Keywords: ANN (Artificial NeuralNetworks); CNN (ConvolutionNeural Networks); Optimization; SGD (Stochastic Gradient Descent); RmsProp (Root mean Square propagation); Adam (Adaptive moment estimation); AdaGrad (Adaptive Gradient)