Resampling Strategies Research Articles

MinoritySalMix and adaptive semantic weight compensation for long-tailed classification

In real-world datasets, the widespread presence of a long-tailed distribution often leads models to become overly biased towards majority class samples while ignoring minority class samples. We propose a strategy called MASW (MinoritySalMix and adaptive semantic weight) to improve this problem. First, we propose a data augmentation method called MinoritySalMix (minority-saliency-mixing), which uses significance detection techniques to select significant regions from minority class samples as cropping regions and paste them into the same regions of majority class samples to generate brand new samples, thereby amplifying images containing important regions of minority class samples. Second, in order to make the label value information of the newly generated samples more consistent with the image content of the newly generated samples, we propose an adaptive semantic compensation factor. This factor provides more label value compensation for minority samples based on the different cropping areas, thereby making the new label values closer to the content of the newly generated samples. Improve model performance by generating more accurate new label value information. Finally, considering that some current re-sampling strategies generally lack flexibility in handling class sampling weight allocation and frequently require manual adjustment. We designed an adaptive weight function and incorporated it into the re-sampling strategy to achieve better sampling. The experimental results on three long-tailed datasets show that our method can effectively improve the performance of the model and is superior to most advanced long-tailed methods. Furthermore, we extended MinoritySalMix’s strategy to three balanced datasets for experimentation, and the results indicated that our method surpassed several advanced data augmentation techniques.

A human-in-the-loop ensemble fusion framework for road crash prediction: coping with imbalanced heterogeneous data from the driver-vehicle-environment system

ABSTRACT Road accidents are an inevitable aspect of daily life, and predicting crashes is crucial for minimizing disruptions and advancing intelligent transportation technologies. This study aims to design an ensemble fusion decision system using various base classifiers and a meta-classifier to improve crash prediction efficiency within the driver-vehicle-environment system. We adopted a data-driven strategy to analyze four categories of features—driver demographics, vehicle telemetry, driver inputs, and environmental conditions—collected from a driving simulator. Optimized modeling strategies using AdaBoost, XGBoost, GBM, LightGBM, and CatBoost were implemented. Moreover, statistical logit models were also used to assess the likelihood of crashes and the correlations among key variables. Furthermore, three resampling strategies, SMOTE-TL, SMOTE-ENN, and ADASYN, were employed to address class imbalance. The best performance was achieved with GBM, XGBoost, and AdaBoost as base classifiers, SMOTE-TL for balancing, and CatBoost as the meta-classifier, with 89.78% precision, 95.69% recall, and 92.64% F1-score.

PVF-10: A high-resolution unmanned aerial vehicle thermal infrared image dataset for fine-grained photovoltaic fault classification

Accurate identification of faulty photovoltaic (PV) modules is crucial for the effective operation and maintenance of PV systems. Deep learning (DL) algorithms exhibit promising potential for classifying PV fault (PVF) from thermal infrared (TIR) images captured by unmanned aerial vehicle (UAV), contingent upon the availability of extensive and high-quality labeled data. However, existing TIR PVF datasets are limited by low image resolution and incomplete coverage of fault types. This study proposes a high-resolution TIR PVF dataset with 10 classes, named PVF-10, comprising 5579 cropped images of PV panels collected from 8 PV power plants. These classes are further categorized into two groups according to the repairability of PVF, with 5 repairable and 5 irreparable classes each. Additionally, the circuit mechanisms underlying the TIR image features of typical PVF types are analyzed, supported by high-resolution images, thereby providing comprehensive information for PV operators. Finally, five state-of-the-art DL algorithms are trained and validated based on the PVF-10 dataset using three levels of resampling strategy. The results show that the overall accuracy (OA) of these algorithms exceeds 83%, with the highest OA reaching 93.32%. Moreover, the preprocessing procedure involving resampling and padding strategies are beneficial for improving PVF classification accuracy using PVF-10 datasets. The developed PVF-10 dataset is expected to stimulate further research and innovation in PVF classification.

Appraising protein conformational changes by resampling time-resolved serial x-ray crystallography data.

With the development of serial crystallography at both x-ray free electron laser and synchrotron radiation sources, time-resolved x-ray crystallography is increasingly being applied to study conformational changes in macromolecules. A successful time-resolved serial crystallography study requires the growth of microcrystals, a mechanism for synchronized and homogeneous excitation of the reaction of interest within microcrystals, and tools for structural interpretation. Here, we utilize time-resolved serial femtosecond crystallography data collected from microcrystals of bacteriorhodopsin to compare results from partial occupancy structural refinement and refinement against extrapolated data. We illustrate the domain wherein the amplitude of refined conformational changes is inversely proportional to the activated state occupancy. We illustrate how resampling strategies allow coordinate uncertainty to be estimated and demonstrate that these two approaches to structural refinement agree within coordinate errors. We illustrate how singular value decomposition of a set of difference Fourier electron density maps calculated from resampled data can minimize phase bias in these maps, and we quantify residual densities for transient water molecules by analyzing difference Fourier and Polder omit maps from resampled data. We suggest that these tools may assist others in judging the confidence with which observed electron density differences may be interpreted as functionally important conformational changes.

A novel generative adversarial network for improving crash severity modeling with imbalanced data

Traffic crash data is often greatly imbalanced with the majority of non-fatal crashes and only a small number of fatal crashes. Such data imbalance issue poses a challenge for crash severity modelling, especially for classifying and interpreting fatal crashes with very limited samples. To address the data imbalance issues, the data resampling techniques are commonly used methods to rebalance the number of samples among all categories of the dataset, such as under-sampling and over-sampling techniques. However, it is challenging for most traditional and existing deep learning-based resampling methods, e.g., synthetic minority oversampling technique (SMOTE) and Generative Adversarial Networks (GAN), to handle both continuous and discrete risk factors in traffic crash datasets, since they are built upon by smooth and continuous functions which are not applicable for processing discrete variables. Though some resampling methods are capable of handling both continuous and discrete variables, they may struggle with mode collapse issues associated with sparse discrete risk factors so that the diversity of the underlying data distribution can not be captured due to oversampling repetitive and similar samples. To address the aforementioned issues, the current study proposes a traffic crash data generation method based on the Conditional Tabular GAN (CTGAN) to rebalance crash datasets for improving performance of crash severity classification and interpretation. The designed experiments are conducted to evaluate contributions of the synthetic data for improving crash severity classification, the distribution consistency between synthetic and benchmark datasets, and the parameter recovery (i.e., the accuracy of parameter estimation and probability prediction) for various resampling strategies. A 4-year real-world dataset collected in Washington State, U.S., and Monte Carlo simulations are utilized for demonstrating the designed experiments. The results indicate that crash severity modeling using synthetic data generated by the mix-resampling of CTGAN and random under-sampling (CTGAN-RU) outperforms all baseline methods. In addition, the proposed deep generative method demonstrates the capability in maintaining distribution consistency and achieving accurate parameter recovery. This study can provide valuable insights for traffic safety researchers and engineers into crash severity modeling, especially when handling imbalanced crash data of various types.

The effects of pension on depressive symptoms in Chinese older adults: A moderated multiple mediator model.

This study investigates the impact of pension on depressive symptoms among Chinese older adults. Additional effort is made to test the mediating effect of multidimensional downward intergenerational support and the moderating effect of age on this relationship. A total of 1828 Chinese older community-dwellers who met our inclusion criteria are drawn from the 2018 China Health and Retirement Longitudinal Study. Multivariate regression modeling is applied to analyze the effect of pensions on depressive symptoms of older adults. Additionally, bootstrap method with resampling strategies is used to estimate the mediating effect of three dimensions of downward intergenerational support (instrumental, emotional, and financial support). Further, Johnson-Neyman technique is employed to analysis and visualize the moderating effect of age. The findings reveal a significant inverse relationship between pension levels and depressive symptoms (B=-6.664, SE=2.826, p<0.05). The analysis shows that downward intergenerational emotional support (B=-0.195, Boot SE=0.103, 95% Boot CI [-0.404, -0.003]) serves as a partial mediator in this relationship. Furthermore, the results highlight the moderating role of age in the linkage between pension and depressive symptoms (B=0.065, SE=0.039, p<0.1). This investigation is pioneering in simultaneously assessing the mediating role of multidimensional downward intergenerational support and the moderating effect of age in the context of pension and depressive symptoms. The study underscores the necessity of an interdisciplinary approach in devising comprehensive intervention strategies. These should encompass pension policy consultation, respite services, and other crucial elements aimed at mitigating the severity or reducing the risk of depressive symptoms among the older adults.

Impact of Nature of Medical Data on Machine and Deep Learning for Imbalanced Datasets: Clinical Validity of SMOTE Is Questionable

Dataset imbalances pose a significant challenge to predictive modeling in both medical and financial domains, where conventional strategies, including resampling and algorithmic modifications, often fail to adequately address minority class underrepresentation. This study theoretically and practically investigates how the inherent nature of medical data affects the classification of minority classes. It employs ten machine and deep learning classifiers, ranging from ensemble learners to cost-sensitive algorithms, across comparably sized medical and financial datasets. Despite these efforts, none of the classifiers achieved effective classification of the minority class in the medical dataset, with sensitivity below 5.0% and area under the curve (AUC) below 57.0%. In contrast, the similar classifiers applied to the financial dataset demonstrated strong discriminative power, with overall accuracy exceeding 95.0%, sensitivity over 73.0%, and AUC above 96.0%. This disparity underscores the unpredictable variability inherent in the nature of medical data, as exemplified by the dispersed and homogeneous distribution of the minority class among other classes in principal component analysis (PCA) graphs. The application of the synthetic minority oversampling technique (SMOTE) introduced 62 synthetic patients based on merely 20 original cases, casting doubt on its clinical validity and the representation of real-world patient variability. Furthermore, post-SMOTE feature importance analysis, utilizing SHapley Additive exPlanations (SHAP) and tree-based methods, contradicted established cerebral stroke parameters, further questioning the clinical coherence of synthetic dataset augmentation. These findings call into question the clinical validity of the SMOTE technique and underscore the urgent need for advanced modeling techniques and algorithmic innovations for predicting minority-class outcomes in medical datasets without depending on resampling strategies. This approach underscores the importance of developing methods that are not only theoretically robust but also clinically relevant and applicable to real-world clinical scenarios. Consequently, this study underscores the importance of future research efforts to bridge the gap between theoretical advancements and the practical, clinical applications of models like SMOTE in healthcare.

Analysing the Relationship between Spatial Resolution, Sharpness and Signal-to-Noise Ratio of Very High Resolution Satellite Imagery Using an Automatic Edge Method

Assessing the performance of optical imaging systems is crucial to evaluate their capability to satisfy the product requirements for an Earth Observation (EO) mission. In particular, the evaluation of image quality is undoubtedly one of the most important, critical and problematic aspects of remote sensing. It involves not only pre-flight analyses, but also continuous monitoring throughout the operational lifetime of the observing system. The Ground Sampling Distance (GSD) of the imaging system is often the only parameter used to quantify its spatial resolution, i.e., its capability to resolve objects on the ground. In practice, this feature is also heavily influenced by other image quality parameters such as the image sharpness and Signal-to-Noise Ratio (SNR). However, these last two aspects are often analysed separately, using unrelated methodologies, complicating the image quality assessment and posing standardisation issues. To this end, we expanded the features of our Automatic Edge Method (AEM), which was originally developed to simplify and automate the estimate of sharpness metrics, to also extract the image SNR. In this paper we applied the AEM to a wide range of optical satellite images characterised by different GSD and Pixel Size (PS) with the objective to explore the nature of the relationship between the components of overall image quality (image sharpness, SNR) and product geometric resampling (expressed in terms of GSD/PS ratio). Our main objective is to quantify how the sharpness and the radiometric quality of an image product are affected by different product geometric resampling strategies, i.e., by distributing imagery with a PS larger or smaller than the GSD of the imaging system. The AEM allowed us to explore this relationship by relying on a vast amount of data points, which provide a robust statistical significance to the results expressed in terms of sharpness metrics and SNR means. The results indicate the existence of a direct relationship between the product geometric resampling and the overall image quality, and also highlight a good degree of correlation between the image sharpness and SNR.

Correlation Between Life Satisfaction and Symptoms of ADHD in Dental Students: The Mediation of Resilience.

This study explores the correlation between life satisfaction and ADHD symptoms. It also discusses whether resilience mediates the correlation between ADHD symptoms and life satisfaction. We surveyed 297 dental students. A total of 291 completed a self-report scale consisting of the Adult ADHD Self-Report Scale, Wender Utah Rating Scale, Life Satisfaction Scale, and Conner-Davidson Resilience Scale. The study used hierarchical linear regression analysis, resampling, and asymptotic strategies for data processing. The ADHD screening results of the self-report scale were positive for 6.87% of the students. This positive rate differed among participants of diverse ages and varying paternal education levels. ADHD symptoms were negatively correlated with life satisfaction and resilience. Life satisfaction was observably positively associated with resilience. Resilience serves as a mediating role between life satisfaction and the two symptoms of ADHD. Resilience intervention programs can enhance the life satisfaction of dental students, especially those with positive ADHD symptoms screening.

Using data resampling and category weight adjustment to solve sample imbalance

The purpose of this thesis is to investigate the application of artificial intelligence and machine learning in solving the sample imbalance problem. The sample imbalance problem refers to the phenomenon that the number of different categories of samples in the training data varies greatly, resulting in the poor performance of traditional machine learning algorithms on a few categories of samples. To address this problem, this paper proposes a new approach combining data resampling and category weight adjustment strategies. First, the sample distribution of the dataset is adjusted by undersampling and oversampling techniques to balance the number of samples from different categories. Then, during the model training process, different weights are assigned to the samples of different categories so that the model pays more attention to the samples of a few categories. The experimental results show that the method achieves significant performance improvement on multiple datasets. In addition, this paper compares other commonly used methods for solving the sample imbalance problem and analyzes and discusses them in detail. Finally, this study offers a practical solution to the problem of sample imbalance and provides guidance for research in related fields.

Customer churn prediction in imbalanced datasets with resampling methods: A comparative study

Customer churn presents a significant challenge for businesses in the era of subscription-based services because retaining customers plays a key role in sustained growth. Existing techniques for automatic churn prediction suffer from a primary challenge inherent in datasets as their significant disproportion between majority and minority classes, which may result in model bias favoring the dominant class. This study presents a comprehensive analysis of Customer Churn Prediction (CCP) with a focus on three public highly imbalanced datasets. The explored datasets span diverse business sectors, including telecommunications, online retail, and banking. We employ a comparative analysis regarding fourteen distinct classification methods considering popular resampling strategies, namely the Synthetic Minority Over-sampling Technique (SMOTE) and the Adaptive Synthetic Sampling (ADASYN). In particular, we investigate a specific configuration that combines a novel two-phase resampling method predicated on both clustering and ensemble techniques in conjunction with Long Short-Term Memory (LSTM) networks. Our findings demonstrate competitive effectiveness, underscoring its potential for effective imbalance correction by further enhancing prediction accuracy. Achieved results suggest that in almost all instances, the integrated approach outperforms the standalone methods across different scenarios in the three datasets, particularly in terms of the Area Under the Curve (AUC). This research represents a significant contribution to the field of churn prediction for addressing class imbalance.

Hierarchical Knowledge Guided Learning for Real-World Retinal Disease Recognition.

In the real world, medical datasets often exhibit a long-tailed data distribution (i.e., a few classes occupy the majority of the data, while most classes have only a limited number of samples), which results in a challenging long-tailed learning scenario. Some recently published datasets in ophthalmology AI consist of more than 40 kinds of retinal diseases with complex abnormalities and variable morbidity. Nevertheless, more than 30 conditions are rarely seen in global patient cohorts. From a modeling perspective, most deep learning models trained on these datasets may lack the ability to generalize to rare diseases where only a few available samples are presented for training. In addition, there may be more than one disease for the presence of the retina, resulting in a challenging label co-occurrence scenario, also known as multi-label, which can cause problems when some re-sampling strategies are applied during training. To address the above two major challenges, this paper presents a novel method that enables the deep neural network to learn from a long-tailed fundus database for various retinal disease recognition. Firstly, we exploit the prior knowledge in ophthalmology to improve the feature representation using a hierarchy-aware pre-training. Secondly, we adopt an instance-wise class-balanced sampling strategy to address the label co-occurrence issue under the long-tailed medical dataset scenario. Thirdly, we introduce a novel hybrid knowledge distillation to train a less biased representation and classifier. We conducted extensive experiments on four databases, including two public datasets and two in-house databases with more than one million fundus images. The experimental results demonstrate the superiority of our proposed methods with recognition accuracy outperforming the state-of-the-art competitors, especially for these rare diseases.

Resampling Strategy in Sequential Monte Carlo for Constrained Sampling Problems

Sequential Monte Carlo (SMC) methods are a class of Monte Carlo methods that are used to obtain random samples of a high dimensional random variable in a sequential fashion. Many problems encountered in applications often involve different types of constraints. These constraints can make the problem much more challenging. In this paper, we formulate a general framework of using SMC for constrained sampling problems based on forward and backward pilot resampling strategies. We review some existing methods under the framework and develop several new algorithms. It is noted that all information observed or imposed on the underlying system can be viewed as constraints. Hence the approach outlined in this paper can be useful in many applications.

A prospective association between social isolation and cognitive performance among older adults in Europe: the role of loneliness and poor oral health

Objectives The relationship between social isolation, loneliness, and tooth loss and cognition in older people is poorly understood. We examine how social isolation and cognitive performance are associated prospectively among older adults, as well as how tooth loss and loneliness are related to this association. Methods Using data from 26,168 participants aged ≥50 from the Survey of Health, Ageing and Retirement in Europe (SHARE), we explored the association between social isolation, loneliness, tooth loss and cognition. We used bootstrapping with resampling strategies for testing a moderated mediating model. Results Higher social isolation was associated with poorer cognitive performance (B = −0.20, 95% CI = −0.03, −0.01; R 2 =0.60), an association mediated by the respondent’s number of missing teeth (B = −0.001, 95% CI = −0.002, −0.001). Higher levels of social isolation were associated with a greater number of missing teeth, and a higher number of missing teeth was linked with poorer cognition. We also found that loneliness moderated the relationship between social isolation and both the number of missing teeth (B = −0.11, p = 0.047) and cognitive performance. Conclusion In later life, social isolation and loneliness are associated with shoddy oral health and poor cognitive status. Clinicians and policymakers should be aware of both the association between social isolation and feelings of loneliness on dentition and oral health and their relationship to the cognitive status of older adults.

Efficient Fusion Decision System for Predicting Road Crash Events: A Comparative Simulator Study for Imbalance Class Handling

Road crash events are a fact of life. Although significant progress have been made in adopting machine learning techniques for analyzing road crashes, there has been limited emphasis on evaluating crash events within data fusion systems. The primary purpose of this study is to outline and validate a comparative safety analysis of an ensemble fusion system founded on the use of different base classifiers and a meta-classifier to procure more efficient crash prediction. Three categories of features namely vehicle-telemetry, driver-inputs and environmental-conditions have been collected using a driving-simulator in order to identify the crash strongest precursors through feature extraction technique. Furthermore, optimized strategies using AdaBoost, XGBoost, RF, GBM, LightGBM, CatBoost and KNN techniques were implemented to establish effective predictions within a fusion-based system. To ensure that the proposed system provide superior decisions given the infrequent nature of crash events, an imbalance-learning approach was conducted based on three resampling strategies: over-sampling, under-sampling and SMOTE-Tomek-Links. The findings depict that the superior performance has been attained when adopting LightGBM, CatBoost and KNN as base classifiers along with SMOTE-TL as balancing technique and XGBoost as meta-classifier with 89.19% precision, 96.77% recall and 92.83% f1-score. To our knowledge, there has been a limited interest, if not at all, at endorsing a fusion-based system examining the impact of real-time features' combinations on the prediction of road crashes while providing a critical analysis of class-imbalance. Overall, the findings emphasized the relevance of the explanatory features and can be endorsed in designing efficient intelligent transportation systems.

Conditional Particle Filters with Bridge Backward Sampling

Conditional particle filters (CPFs) with backward/ancestor sampling are powerful methods for sampling from the posterior distribution of the latent states of a dynamic model such as a hidden Markov model. However, the performance of these methods deteriorates with models involving weakly informative observations and/or slowly mixing dynamics. Both of these complications arise when sampling finely time-discretized continuous-time path integral models, but can occur with hidden Markov models too. Multinomial resampling, which is commonly employed with CPFs, resamples excessively for weakly informative observations and thereby introduces extra variance. Furthermore, slowly mixing dynamics render the backward/ancestor sampling steps ineffective, leading to degeneracy issues. We detail two conditional resampling strategies suitable for the weakly informative regime: the so-called “killing” resampling and the systematic resampling with mean partial order. To avoid the degeneracy issues, we introduce a generalization of the CPF with backward sampling that involves auxiliary “bridging” CPF steps that are parameterized by a blocking sequence. We present practical tuning strategies for choosing an appropriate blocking. Our experiments demonstrate that the CPF with a suitable resampling and the developed “bridge backward sampling” can lead to substantial efficiency gains in the weakly informative and slow mixing regime. Supplementary materials for this article are available online.

Predicting additive manufacturing defects with robust feature selection for imbalanced data

Promptly predicting defects during an additive manufacturing process using only copious log data provides many advantages, albeit with computational limitations. We focus on predicting defects during electron beam melting with the black box nature of the manufacturing machine. For an accurate prediction of defects, which are rare ( < 2 % ), we extract temporal information to track abnormalities and formulate a feature selection algorithm that maximizes the expected value of a cost-sensitive accuracy. Correct identification of features responsible for the defects increases predictive power and informs manufacturers of potential corrective/preventive actions for process improvement. We solve the feature selection through resampling strategies integrated with ensemble procedures to handle data uncertainty and imbalance. Exploiting data uncertainty in our search leads to finding robust features with consistent predictive power. Our proposed methodology shows a 43% improvement in predicting defects (recall) without losing precision. Beyond additive manufacturing, this methodology has general application for rare-event prediction and imbalanced datasets.

Adaptive Subspace Optimization Ensemble Method for High-Dimensional Imbalanced Data Classification.

It is hard to construct an optimal classifier for high-dimensional imbalanced data, on which the performance of classifiers is seriously affected and becomes poor. Although many approaches, such as resampling, cost-sensitive, and ensemble learning methods, have been proposed to deal with the skewed data, they are constrained by high-dimensional data with noise and redundancy. In this study, we propose an adaptive subspace optimization ensemble method (ASOEM) for high-dimensional imbalanced data classification to overcome the above limitations. To construct accurate and diverse base classifiers, a novel adaptive subspace optimization (ASO) method based on adaptive subspace generation (ASG) process and rotated subspace optimization (RSO) process is designed to generate multiple robust and discriminative subspaces. Then a resampling scheme is applied on the optimized subspace to build a class-balanced data for each base classifier. To verify the effectiveness, our ASOEM is implemented based on different resampling strategies on 24 real-world high-dimensional imbalanced datasets. Experimental results demonstrate that our proposed methods outperform other mainstream imbalance learning approaches and classifier ensemble methods.

An Oversampling Mechanism for Multimajority Datasets using SMOTE and Darwinian Particle Swarm Optimisation

Data skewness continues to be one of the leading factors which adversely impacts the machine learning algorithms performance. An approach to reduce this negative effect of the data variance is to pre-process the former dataset with data level resampling strategies. Resampling strategies have been seen in two forms, oversampling and undersampling. An oversampling strategy is proposed in this article for tackling multiclass imbalanced datasets. This proposed approach optimises the state-of-the-art oversampling technique SMOTE with the Darwinian Particle Swarm Optimization technique. This proposed method DOSMOTE generates synthetic optimised samples for balancing the datasets. This strategy will be more effective on multimajority datasets. An experimental study is performed on peculiar multimajority datasets to measure the effectiveness of the proposed approach. As a result, the proposed method produces promising results when compared to the conventional oversampling strategies.

5-Year progression prediction of endplate defects: Utilizing the EDPP-Flow convolutional neural network based on unbalanced data

BackgroundLumbar disc degeneration (LDD) is considered as one of the main causes of low back pain. For clinical diagnosis of LDD, magnetic resonance imaging (MRI) is commonly used. Schmorl's node, high intensity zone (HIZ), Modic changes, and other MRI biomarkers of intervertebral disc (IVD) degeneration are also associated with low back pain. However, the progression and natural history of these features are unclear and there is limited predictive capacity with MRI. PurposeWe aim to establish and validate a deep learning pipeline, EDPP-Flow, for the 5-year progression prediction of Schmorl's node, HIZ, and Modic changes, based on clinical MRIs. Materials and methodsAn MRI dataset developed on 1152 volunteers was used in this study. For each volunteer, two MRI scans, at baseline and 5-year follow-up, were collected and pathology labels were annotated as present or absent (with/without pathology) by two specialists with over 10 years of clinical experience. Our pipeline contained the published MRI-SegFlow and state-of-the-art convolutional neural network for progression prediction of endplate defects. The label distribution of the dataset is unbalanced, where the number of present samples was much smaller than absent samples. The resampling and data augmentation strategies were adopted to increase the number of present samples in the training process and balance the influence of different samples on the model, which can improve the prediction accuracy. ResultsOur pipeline achieved high weighted accuracy, sensitivity, and specificity for progression prediction of Schmorl's node (89.46 ± 3.71%, 89.19 ± 2.70%, 89.72 ± 2.42%), HIZ (91.75 ± 2.48%, 93.07 ± 3.96%, 90.43 ± 2.51%), and Modic changes (87.51 ± 2.23%, 87.93 ± 1.72%, 87.10 ± 1.99%), on the unbalanced dataset (present sample's percentages of the 3 pathologies above were 4.3%, 11.7%, and 6.7%). ConclusionWe developed and validated a deep learning pipeline, for the progression prediction of endplate defects, which showed high prediction accuracy on unbalanced data. The method has significant potential for clinical implementation.