Training Deep Neural Networks Research Articles

Genotype sampling for deep-learning assisted experimental mapping of a combinatorially complete fitness landscape.

Experimental characterization of fitness landscapes, which map genotypes onto fitness, is important for both evolutionary biology and protein engineering. It faces a fundamental obstacle in the astronomical number of genotypes whose fitness needs to be measured for any one protein. Deep learning may help to predict the fitness of many genotypes from a smaller neural network training sample of genotypes with experimentally measured fitness. Here I use a recently published experimentally mapped fitness landscape of more than 260,000 protein genotypes to ask how such sampling is best performed. I show that multilayer perceptrons, recurrent neural networks (RNNs), convolutional networks, and transformers, can explain more than 90 percent of fitness variance in the data. In addition, 90 percent of this performance is reached with a training sample comprising merely ≈103 sequences. Generalization to unseen test data is best when training data is sampled randomly and uniformly, or sampled to minimize the number of synonymous sequences. In contrast, sampling to maximize sequence diversity or codon usage bias reduces performance substantially. These observations hold for more than one network architecture. Simple sampling strategies may perform best when training deep learning neural networks to map fitness landscapes from experimental data. The fitness landscape data analyzed here is publicly available as described previously (Papkou, et al., 2023). All code used to analyze this landscape is publicly available at https://github.com/andreas-wagner-uzh/fitness_landscape_sampling. Supplementary data are available at Bioinformatics online.

Hierarchical Training of Deep Neural Networks Using Early Exiting.

Deep neural networks (DNNs) provide state-of-the-art accuracy for vision tasks, but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime, and privacy concerns. In this study, a novel hierarchical training method for DNNs is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime, and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that, due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud, and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime for VGG-16 and ResNet-18 architectures by 29% and 61% in CIFAR-10 classification and by 25% and 81% in Tiny ImageNet classification, respectively, when the communication with the cloud is done over a low bit rate channel. This gain in the runtime is achieved, while the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy DNNs on sensor-holding low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

VLSI architecture of stochastic genetic algorithm for real time training of deep neural network

VLSI architecture of stochastic genetic algorithm for real time training of deep neural network

Machine-Learning driven STM images prediction of doped/defective graphene: Towards optimized tools for 2D nanomaterials characterization

Scanning tunneling microscopy (STM) is a key characterization technique that allows for visualization of specific features at nanostructures, given its ability to reveal changes in local charge densities in doping or defective sites. Besides the experimental acquisition of STM data from real samples, there is the theoretical route, which allows for STM images simulation from ab-initio calculations on defined nanostructures. This work presents an alternative route for theoretical STM image prediction by means of Machine Learning (ML) methods, based on the training of deep convolutional neural networks using simulated STM data. The ML-based STM predictions for defective, B- and N-doped graphene quantum dots are compared to ab-initio STM simulations, in terms of accuracy of structural rearrangements, charge density localizations, and computational resources requirements. The results demonstrate that STM images predicted by ML-methods are accurate at doping sites, whereas they show good similarity to simulations of large structural vacancies. This work represents a novel alternative for the use of ML methods in image-based characterization routines of doped/defective graphene, and potentially, for other 2-dimensional nanomaterials.

Enhancing the automatic facies classification of Brazilian presalt acoustic image logs with SwinV2-Unet: Leveraging transfer learning and confident learning

Facies classification of image logs plays a vital role in reservoir characterization, especially in the heterogeneous and anisotropic carbonate formations of the Brazilian presalt region. Although manual classification remains the industry standard for handling the complexity and diversity of image logs, it has the notable disadvantages of being time consuming, labor intensive, subjective, and nonrepeatable. Recent advancements in machine learning offer promising solutions for automation and acceleration. However, previous attempts to train deep neural networks for facies identification have struggled to generalize to new data due to insufficient labeled data and the inherent intricacy of image logs. In addition, human errors in manual labels further hinder the performance of trained models. To overcome these challenges, we develop adopting the state-of-the-art SwinV2-Unet to provide depthwise facies classification for Brazilian presalt acoustic image logs. The training process incorporates transfer learning to mitigate overfitting and confident learning to address label errors. Through a k-fold cross-validation experiment, with each fold spanning more than 350 m, we achieve an impressive macro F1 score of 0.90 for out-of-sample predictions. This significantly surpasses the previous model modified from the widely recognized U-Net, which provides a macro F1 score of 0.68. These findings highlight the effectiveness of the used enhancements, including the adoption of an improved neural network and an enhanced training strategy. Moreover, our SwinV2-Unet enables a highly efficient and accurate facies analysis of the complex yet informative image logs, significantly advancing our understanding of hydrocarbon reservoirs, saving human effort, and improving productivity.

Improved weight initialization for deep and narrow feedforward neural network

Appropriate weight initialization settings, along with the ReLU activation function, have become cornerstones of modern deep learning, enabling the training and deployment of highly effective and efficient neural network models across diverse areas of artificial intelligence. The problem of “dying ReLU,” where ReLU neurons become inactive and yield zero output, presents a significant challenge in the training of deep neural networks with ReLU activation function. Theoretical research and various methods have been introduced to address the problem. However, even with these methods and research, training remains challenging for extremely deep and narrow feedforward networks with ReLU activation function. In this paper, we propose a novel weight initialization method to address this issue. We establish several properties of our initial weight matrix and demonstrate how these properties enable the effective propagation of signal vectors. Through a series of experiments and comparisons with existing methods, we demonstrate the effectiveness of the novel initialization method.

Using an Improved Visual Geometry Group Neural Network for Skin Cancer Detection

Over time, deep learning has become more accurate and efficient at dealing with more complex risks. However, training deep neural networks remains a daunting problem and challenges remain, possibly due to several issues, including relevance, generalizability, and training time. Producing specific processors to manipulate artificial networks and reduce task-specific network learning time is natural, but the challenge of fit and generalizability remains undisputed. These networks have been used to detect and diagnose skin cancer, one of the most dangerous types of cancer caused by DNA damage. Automatic recognition of skin cancer is important to help doctors detect the disease in its early stages. In this research, the proposed vgg16 method was used to classify benign and malignant medical images. Image processing was used, where Gaussian noise and salt and pepper noise were added to increase the data, and filters for the arithmetic mean and median were added for each noise. The network was implemented using the Keras interface. Appling suggest model shown that the height accuracy 0.86 for train and accuracy 0.92 for test

Random feedback alignment algorithms to train neural networks: why do they align?

Feedback alignment algorithms are an alternative to backpropagation to train neural networks, whereby some of the partial derivatives that are required to compute the gradient are replaced by random terms. This essentially transforms the update rule into a random walk in weight space. Surprisingly, learning still works with those algorithms, including training of deep neural networks. The performance of FA is generally attributed to an alignment of the update of the random walker with the true gradient—the eponymous gradient alignment—which drives an approximate gradient descent. The mechanism that leads to this alignment remains unclear, however. In this paper, we use mathematical reasoning and simulations to investigate gradient alignment. We observe that the feedback alignment update rule has fixed points, which correspond to extrema of the loss function. We show that gradient alignment is a stability criterion for those fixed points. It is only a necessary criterion for algorithm performance. Experimentally, we demonstrate that high levels of gradient alignment can lead to poor algorithm performance and that the alignment is not always driving the gradient descent.

A Theoretical View of Linear Backpropagation and its Convergence.

Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by (Guo et al. 2020). Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al. (2020) paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments.

Arterial Input Function (AIF) Correction Using AIF Plus Tissue Inputs with a Bi-LSTM Network

Background: The arterial input function (AIF) is vital for myocardial blood flow quantification in cardiac MRI to indicate the input time–concentration curve of a contrast agent. Inaccurate AIFs can significantly affect perfusion quantification. Purpose: When only saturated and biased AIFs are measured, this work investigates multiple ways of leveraging tissue curve information, including using AIF + tissue curves as inputs and optimizing the loss function for deep neural network training. Methods: Simulated data were generated using a 12-parameter AIF mathematical model for the AIF. Tissue curves were created from true AIFs combined with compartment-model parameters from a random distribution. Using Bloch simulations, a dictionary was constructed for a saturation-recovery 3D radial stack-of-stars sequence, accounting for deviations such as flip angle, T2* effects, and residual longitudinal magnetization after the saturation. A preliminary simulation study established the optimal tissue curve number using a bidirectional long short-term memory (Bi-LSTM) network with just AIF loss. Further optimization of the loss function involves comparing just AIF loss, AIF with compartment-model-based parameter loss, and AIF with compartment-model tissue loss. The optimized network was examined with both simulation and hybrid data, which included in vivo 3D stack-of-star datasets for testing. The AIF peak value accuracy and ktrans results were assessed. Results: Increasing the number of tissue curves can be beneficial when added tissue curves can provide extra information. Using just the AIF loss outperforms the other two proposed losses, including adding either a compartment-model-based tissue loss or a compartment-model parameter loss to the AIF loss. With the simulated data, the Bi-LSTM network reduced the AIF peak error from −23.6 ± 24.4% of the AIF using the dictionary method to 0.2 ± 7.2% (AIF input only) and 0.3 ± 2.5% (AIF + ten tissue curve inputs) of the network AIF. The corresponding ktrans error was reduced from −13.5 ± 8.8% to −0.6 ± 6.6% and 0.3 ± 2.1%. With the hybrid data (simulated data for training; in vivo data for testing), the AIF peak error was 15.0 ± 5.3% and the corresponding ktrans error was 20.7 ± 11.6% for the AIF using the dictionary method. The hybrid data revealed that using the AIF + tissue inputs reduced errors, with peak error (1.3 ± 11.1%) and ktrans error (−2.4 ± 6.7%). Conclusions: Integrating tissue curves with AIF curves into network inputs improves the precision of AI-driven AIF corrections. This result was seen both with simulated data and with applying the network trained only on simulated data to a limited in vivo test dataset.

Landslide susceptibility assessment using deep learning considering unbalanced samples distribution

Landslide susceptibility assessment (LSA) is fundamental for managing landslide geological disasters. This study presents a deep learning approach (DNN-MSFM) designed to enhance LSA modeling, particularly addressing limitations caused by the unbalanced distribution of data samples in applied datasets. DNN-MSFM approach combines a deep neural network (DNN) and a mean squared false misclassification loss function (MSFM) to handle unbalanced samples from the algorithmic perspective. The model's performance was evaluated on an unbalanced dataset containing mapping units' records of 293 landslide samples and 653 non-landslide samples from the Baota District, China. Its effectiveness was assessed through statistical metrics and compared against DNN and Support Vector Machine (SVM) basic models. The results demonstrated a significant performance enhancement from the DNN-MSFM (OverallAccuracy = 0.889 and area under the receiver operating characteristic curve (AUC) = 0.84), indicating its effectiveness in learning the underlying landslide susceptibility features and demonstrating its ability to provide improved predictions even in areas with unbalanced landslide samples. Moreover, the study emphasizes the importance of considering balanced loss functions in training DNN under various imbalance degrees and contributes to expanding the applicability of DNN in LSA modeling. Also, this study builds a foundation for further enhancements of deep learning methods for geological disaster assessments.

Security intrusion monitoring model for Internet of Things (IoT) using sniffing tools on wireless sensor networks

The Internet of Things (IoT) has revolutionized the way devices interact and share data over wireless sensor networks (WSN), enabling seamless connectivity and automation. However, the proliferation of IoT devices has raised serious security and privacy risks concerns due to their inherent vulnerabilities. This paper proposes a model for security intrusion monitoring by analyzing the existing literature and providing insights into the design, implementation, and effective deployment of the proposed model to detect intrusion in IoT using sniffing tools for network traffic analysis in real-time within WSN. The model passively monitors network traffic and identifies anomalous patterns, unauthorized access attempts, and abnormal device behavior. The review findings highlight the significance of the proposed model in enhancing the security of IoT systems. By detecting anomalous behavior and potential security breaches. The model enables timely response and mitigation actions to ensure the confidentiality, integrity and availability (CIA) of IoT devices data. The model includes consideration of network architecture, deployment of intrusion detection algorithms, and establishment of response mechanisms. It identifies various types of security threats, such as unauthorized access attempts, Denial-of-service, Distributed DoS, Brute-force, Heartbleed, Botnet, Inside Infiltration and device tampering, thereby providing response mechanisms that include generating alerts, isolating compromised devices, or blocking suspicious network traffic. The model incorporates a feedback loop to continuously update the detection mechanisms and adapt to evolving security threats in real-time. Series of experiments and simulations to be conducted using various IoT devices and network scenarios to evaluate model effectiveness. The model to comprise of wireless Router, MatLab for Deep Neural Network (DNN) training, Raspberry Pi, Wireshark setup and an array of Internet of Things (IoT) devices. The researcher to use dataset by extracting intrinsic, host-based and time-based attributes from Wireshark Sniffing tool. The datasets generated shall be piped by tshark to an output text file saved as a csv. Under-sampling technique to be used to address class imbalance of datasets. The model shall then be trained using the dataset to be able to detect intrusion in IoTs. The results is expected to demonstrate the model's ability to detect a wide range of security intrusions with high accuracy and minimal false positives. In conclusion, the model offers a proactive approach to safeguard IoT deployment. By leveraging sniffing tools and advanced analysis techniques, the model enhances the detection and response capabilities, enabling efficient protection against emerging threats in IoT. However, challenges associated with the model are identified, including the complexity of network monitoring and potential privacy concerns.

Quantification of cervical spinal stenosis by automated 3D MRI segmentation of spinal cord and cerebrospinal fluid space

Abstract Design Prospective diagnostic study. Objectives Anatomical evaluation and graduation of the severity of spinal stenosis is essential in degenerative cervical spine disease. In clinical practice, this is subjectively categorized on cervical MRI lacking an objective and reliable classification. We implemented a fully-automated quantification of spinal canal compromise through 3D T2-weighted MRI segmentation. Setting Medical Center - University of Freiburg, Germany. Methods Evaluation of 202 participants receiving 3D T2-weighted MRI of the cervical spine. Segments C2/3 to C6/7 were analyzed for spinal cord and cerebrospinal fluid space volume through a fully-automated segmentation based on a trained deep convolutional neural network. Spinal canal narrowing was characterized by relative values, across sever segments as adapted Maximal Canal Compromise (aMCC), and within the index segment as adapted Spinal Cord Occupation Ratio (aSCOR). Additionally, all segments were subjectively categorized by three observers as “no”, “relative” or “absolute” stenosis. Computed scores were applied on the subjective categorization. Results 798 (79.0%) segments were subjectively categorized as “no” stenosis, 85 (8.4%) as “relative” stenosis, and 127 (12.6%) as “absolute” stenosis. The calculated scores revealed significant differences between each category (p ≤ 0.001). Youden’s Index analysis of ROC curves revealed optimal cut-offs to distinguish between “no” and “relative” stenosis for aMCC = 1.18 and aSCOR = 36.9%, and between “relative” and “absolute” stenosis for aMCC = 1.54 and aSCOR = 49.3%. Conclusion The presented fully-automated segmentation algorithm provides high diagnostic accuracy and objective classification of cervical spinal stenosis. The calculated cut-offs can be used for convenient radiological quantification of the severity of spinal canal compromise in clinical routine.

Per-class curriculum for Unsupervised Domain Adaptation in semantic segmentation

AbstractAccurate training of deep neural networks for semantic segmentation requires a large number of pixel-level annotations of real images, which are expensive to generate or not even available. In this context, Unsupervised Domain Adaptation (UDA) can transfer knowledge from unlimited synthetic annotations to unlabeled real images of a given domain. UDA methods are composed of an initial training stage with labeled synthetic data followed by a second stage for feature alignment between labeled synthetic and unlabeled real data. In this paper, we propose a novel approach for UDA focusing the initial training stage, which leads to increased performance after adaptation. We introduce a curriculum strategy where each semantic class is learned progressively. Thereby, better features are obtained for the second stage. This curriculum is based on: (1) a class-scoring function to determine the difficulty of each semantic class, (2) a strategy for incremental learning based on scoring and pacing functions that limits the required training time unlike standard curriculum-based training and (3) a training loss to operate at class level. We extensively evaluate our approach as the first stage of several state-of-the-art UDA methods for semantic segmentation. Our results demonstrate significant performance enhancements across all methods: improvements of up to 10% for entropy-based techniques and 8% for adversarial methods. These findings underscore the dependency of UDA on the accuracy of the initial training. The implementation is available at https://github.com/vpulab/PCCL.

Enhancing Skin Disease Classification: A Novel Approach With Tailored Loss Functions And SMOTE Sumeet Ghumare

Although skin disease like skin cancer is a common health problem worldwide, early identification is crucial to its recovery. By using medical photos, the rise of artificial intelligence (AI) has transformed the detection of skin diseases. Despite the reality that various deep learning models have been created with this goal in mind, class imbalance in multi class classification all poses a serious difficulty and imbalance among the classes. This work offers a novel strategy to reduce class imbalance in the categorization of skin diseases, leading to significant accuracy gains. One of the largest skin cancer datasets currently accessible, it consists of 33,106 dermoscopy photos from seven different kinds of skin diseases as a starting point. The main goal is to enhance skin disease classification algorithms’ performance on this imbalance dataset. We are going to use a two-pronged strategy to combat the class disparity and GAN (Generative adversarial network) Based Oversampling or Meta-learning in order to successfully equalize the representation of each class during training, we first design a balanced Dynamic Mini-Batch Size logic, SMOTE (Synthetic Minority Oversampling Technique) and Augmenter. Second, we provide a new loss function designed to take into account the unique features of the dataset. The outcomes are startling. These findings underscore the efficacy of our hybrid approach in training deep convolutional neural networks for imbalanced skin disease datasets. By combining data-level balancing techniques with a carefully designed loss function at the algorithm level, we address the challenge of among the classes, paving the way for more accurate and reliable skin cancer diagnoses. Key Words: Dermoscopy images, DenseNet169, Convolution neural network, SMOTE (Synthetic Minority Over-sampling Technique, GAN(Generative adversarial network), Mini-Batch Logic, Augmenter

Principal Component Networks: Utilizing Low-Rank Activation Structure to Reduce Parameters Early in Training

Recent works show that overparameterized neural networks contain small subnetworks that exhibit comparable accuracy to the full model when trained in isolation. These results highlight the potential to reduce the computational costs of deep neural network training without sacrificing generalization performance. Initial approaches for finding these small networks relied on expensive multi-round train-and-prune procedures, limiting their practical potential, but more recent work identifies subnetworks using structured pruning techniques early in training. In this article, we study network activations, rather than network weights, and find that hidden layer activations in overparameterized networks exist primarily in subspaces smaller than the actual model width. We further notice that these subspaces can be identified early in training. Based on these observations, we show how to efficiently find small networks that exhibit similar accuracy to their overparameterized counterparts after only a few training epochs. We term these network architectures Principal Component Networks (PCNs). PCNs compress individual layers by retaining only the high variance linear combinations of channels—defined by the principal components of the layer inputs—a key difference from structured pruning techniques which focus on individual channel pruning using localized channel measurements. We evaluate PCNs on CIFAR-10 and ImageNet for VGG and ResNet style architectures and compare against existing methods for subnetwork identification during early training. We find that PCNs consistently reduce parameter counts with little accuracy loss, thus providing the potential to reduce the computational costs of deep neural network training. Beyond model compression, we also connect our observation regarding hidden layer activations to the feature representations learned by neural networks and discuss areas for future work.

Optimizing DNN training with pipeline model parallelism for enhanced performance in embedded systems

Deep Neural Networks (DNNs) have gained widespread popularity in different domain applications due to their dominant performance. Despite the prevalence of massively parallel multi-core processor architectures, adopting large DNN models in embedded systems remains challenging, as most embedded applications are designed with single-core processors in mind. This limits DNN adoption in embedded systems due to inefficient leveraging of model parallelization and workload partitioning. Prior solutions attempt to address these challenges using data and model parallelism. However, they lack in finding optimal DNN model partitions and distributing them efficiently to achieve improved performance.This paper proposes a DNN model parallelism framework to accelerate model training by finding the optimal number of model partitions and resource provisions. The proposed framework combines data and model parallelism techniques to optimize the parallel processing of DNNs for embedded applications. In addition, it implements the pipeline execution of the partitioned models and integrates a task controller to manage the computing resources. The experimental results for image object detection demonstrate the applicability of our proposed framework in estimating the latest execution time and reducing overall model training time by almost 44.87% compared to the baseline AlexNet convolutional neural network (CNN) model.

Physics-informed State-space Neural Networks for transport phenomena

This work introduces Physics-informed State-space neural network Models (PSMs), a novel solution to achieving real-time optimization, flexibility, and fault tolerance in autonomous systems, particularly in transport-dominated systems such as chemical, biomedical, and power plants. Traditional data-driven methods fall short due to a lack of physical constraints like mass conservation; PSMs address this issue by training deep neural networks with sensor data and physics-informing using components’ Partial Differential Equations (PDEs), resulting in a physics-constrained, end-to-end differentiable forward dynamics model. Through two in silico experiments – a heated channel and a cooling system loop – we demonstrate that PSMs offer a more accurate approach than a purely data-driven model. In the former experiment, PSMs demonstrated significantly lower average root-mean-square errors across test datasets compared to a purely data-driven neural network, with reductions of 44 %, 48 %, and 94 % in predicting pressure, velocity, and temperature, respectively.Beyond accuracy, PSMs demonstrate a compelling multitask capability, making them highly versatile. In this work, we showcase two: supervisory control of a nonlinear system through a sequentially updated state-space representation and the proposal of a diagnostic algorithm using residuals from each of the PDEs. The former demonstrates PSMs’ ability to handle constant and time-dependent constraints, while the latter illustrates their value in system diagnostics and fault detection.

Learning with noisy labels using collaborative sample selection and contrastive semi-supervised learning

Learning with noisy labels (LNL) has been extensively studied, with existing approaches typically following a framework that alternates between clean sample selection and semi-supervised learning (SSL). However, this approach has a limitation: the clean set selected by the Deep Neural Network (DNN) classifier, trained through self-training, inevitably contains noisy samples. This mixture of clean and noisy samples leads to misguidance in DNN training during SSL, resulting in impaired generalization performance due to confirmation bias caused by error accumulation in sample selection. To address this issue, we propose a method called Collaborative Sample Selection (CSS), which leverages the large-scale pre-trained model CLIP. CSS aims to remove the mixed noisy samples from the identified clean set. We achieve this by training a 2-Dimensional Gaussian Mixture Model (2D-GMM) that combines the probabilities from CLIP with the predictions from the DNN classifier. To further enhance the adaptation of CLIP to LNL, we introduce a co-training mechanism with a contrastive loss in semi-supervised learning. This allows us to jointly train the prompt of CLIP and the DNN classifier, resulting in improved feature representation, boosted classification performance of DNNs, and reciprocal benefits to our Collaborative Sample Selection. By incorporating auxiliary information from CLIP and utilizing prompt fine-tuning, we effectively eliminate noisy samples from the clean set and mitigate confirmation bias during training. Experimental results on multiple benchmark datasets demonstrate the effectiveness of our proposed method in comparison with the state-of-the-art approaches.

Fine-Grained Self-Supervised Learning with Jigsaw puzzles for medical image classification

Classifying fine-grained lesions is challenging due to minor and subtle differences in medical images. This is because learning features of fine-grained lesions with highly minor differences is very difficult in training deep neural networks. Therefore, in this paper, we introduce Fine-Grained Self-Supervised Learning(FG-SSL) method for classifying subtle lesions in medical images. The proposed method progressively learns the model through hierarchical block such that the cross-correlation between the fine-grained Jigsaw puzzle and regularized original images is close to the identity matrix. We also apply hierarchical block for progressive fine-grained learning, which extracts different information in each step, to supervised learning for discovering subtle differences. Our method does not require an asymmetric model, nor does a negative sampling strategy, and is not sensitive to batch size. We evaluate the proposed fine-grained self-supervised learning method on comprehensive experiments using various medical image recognition datasets. In our experiments, the proposed method performs favorably compared to existing state-of-the-art approaches on the widely-used ISIC2018, APTOS2019, and ISIC2017 datasets.