Ordinal Regression Problem Research Articles

MGScoliosis: Multi-grained scoliosis detection with joint ordinal regression from natural image

Scoliosis is among the most prevalent diseases affecting teenagers. However, traditional scoliosis screening methods often resort to physical examination or radiographic imaging. The two ways both rely on experts with high costs, which are not suitable for wide-range screening. Besides, estimating Cobb angle level only using natural images are challenging. To tackle these issues, we propose a multi-grained scoliosis detection framework by jointly estimating severity level and Cobb angle level of scoliosis from a natural image instead of a radiographic image, which has not been explored before. Specifically, we regard scoliosis estimation as an ordinal regression problem, and transform it into a series of binary classification sub-problems. Besides, we adopt the visual attention network with large kernel attention as the backbone for feature learning, which can model local and global correlations with efficient computations. The feature learning and the ordinal regression is put into an end-to-end framework, in which the two tasks of scoliosis severity level estimation and scoliosis angle level estimation are jointly learned and can contribute to each other. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods as well as human performance, which provides a promising and economical solution to wide-range scoliosis screening. Particularly, our approach achieves accuracies of 94.90% and 79.62% in estimating severity level and Cobb angle level, improving large margins of 4.90% and 12.15% over existing natural image based scoliosis detection performance, respectively. The code is available at https://github.com/RuiChen-stack/MGScoliosis.

Deep Learning-Based Dynamic Region of Interest Autofocus Method for Grayscale Image.

In the field of autofocus for optical systems, although passive focusing methods are widely used due to their cost-effectiveness, fixed focusing windows and evaluation functions in certain scenarios can still lead to focusing failures. Additionally, the lack of datasets limits the extensive research of deep learning methods. In this work, we propose a neural network autofocus method with the capability of dynamically selecting the region of interest (ROI). Our main work is as follows: first, we construct a dataset for automatic focusing of grayscale images; second, we transform the autofocus issue into an ordinal regression problem and propose two focusing strategies: full-stack search and single-frame prediction; and third, we construct a MobileViT network with a linear self-attention mechanism to achieve automatic focusing on dynamic regions of interest. The effectiveness of the proposed focusing method is verified through experiments, and the results show that the focusing MAE of the full-stack search can be as low as 0.094, with a focusing time of 27.8 ms, and the focusing MAE of the single-frame prediction can be as low as 0.142, with a focusing time of 27.5 ms.

CATNet: Convolutional attention and transformer for monocular depth estimation

Monocular depth estimation has received more and more attention due to its wide range of application scenarios. In this paper, we propose a novel simple framework, called CATNet, which treats monocular depth estimation as an ordinal regression problem. At present, in order to obtain higher performance, the research on monocular depth estimation is achieved by increasing the amount of calculation and parameters of the model. Based on this, we propose a novel simple encoder–decoder architecture that aims to reduce the SOTA model parameters and complexity while keeping the depth estimation accuracy as high as possible rather than aiming for extremely lightweight. Meanwhile, in order to further refine the multi-scale information extracted by the encoder, we propose a Multi-dimensional Convolutional Attention (MCA) module. To enhance the extraction of global information for accurate pixel classification, we propose a Dual Attention Transformer (DAT) module to extract global features of images. Furthermore, experimental results on the KITTI and NYU datasets demonstrate that the advantage of our proposed framework is that it achieves almost equivalent depth estimation performance to the current SOTA with fewer parameters and lower complexity. To the best of our knowledge, CATNet is the first work that achieves nearly the same depth estimation accuracy as Transformer-based large model encoders with so few parameters.

Deep Ordinal Regression Framework for No-Reference Image Quality Assessment

Due to the rapid development of deep learning techniques, no-reference image quality assessment (NR-IQA) has achieved significant improvement. NR-IQA aims to predict a real-valued variable for image quality, using the image in question as the sole input. Existing deep learning-based NR-IQA models are formulated as a regression problem and trained by minimising the mean squared error. The error measurement does not consider the relative ordering between different ratings on the quality scale, which consequently affects the efficacy of the model. To account for this problem, we reformulate NR-IQA learning as an ordinal regression problem and propose a simple yet effective framework using deep convolutional neural networks (DCNN) and Transformers. NR-IQA learning is achieved by a deep ordinal loss and using a soft ordinal inference to transform the predicted probabilities to a continuous variable for image quality. Experimental results demonstrate the superiority of our proposed NR-IQA model based on deep ordinal regression. In addition, this framework can be easily extended with various DCNN architectures to build advanced IQA models.

Deciphering Feature Effects on Decision-Making in Ordinal Regression Problems: An Explainable Ordinal Factorization Model

Ordinal regression predicts the objects’ labels that exhibit a natural ordering, which is vital to decision-making problems such as credit scoring and clinical diagnosis. In these problems, the ability to explain how the individual features and their interactions affect the decisions is as critical as model performance. Unfortunately, the existing ordinal regression models in the machine learning community aim at improving prediction accuracy rather than explore explainability. To achieve high accuracy while explaining the relationships between the features and the predictions, we propose a new method for ordinal regression problems, namely the Explainable Ordinal Factorization Model (XOFM). XOFM uses piecewise linear functions to approximate the shape functions of individual features, and renders the pairwise features interaction effects as heat-maps. The proposed XOFM captures the nonlinearity in the main effects and ensures the interaction effects’ same flexibility. Therefore, the underlying model yields comparable performance while remaining explainable by explicitly describing the main and interaction effects. To address the potential sparsity problem caused by discretizing the whole feature scale into several sub-intervals, XOFM integrates the Factorization Machines (FMs) to factorize the model parameters. Comprehensive experiments with benchmark real-world and synthetic datasets demonstrate that the proposed XOFM leads to state-of-the-art prediction performance while preserving an easy-to-understand explainability.

Dwell Time Estimation of Import Containers as an Ordinal Regression Problem

The optimal stacking of import containers in a terminal reduces the reshuffles during the unloading operations. Knowing the departure date of each container is critical for optimal stacking. However, such a date is rarely known because it depends on various attributes. Therefore, some authors have proposed estimation algorithms using supervised classification. Although supervised classifiers can estimate this dwell time, the variable “dwell time” takes ordered values for this problem, suggesting using ordinal regression algorithms. Thus, we have compared an ordinal regression algorithm (selected from 15) against two supervised classifiers (selected from 30). We have set up two datasets with data collected in a container terminal. We have extracted and evaluated 35 attributes related to the dwell time. Additionally, we have run 21 experiments to evaluate both approaches regarding the mean absolute error modified and the reshuffles. As a result, we have found that the ordinal regression algorithm outperforms the supervised classifiers, reaching the lowest mean absolute error modified in 15 (71%) and the lowest reshuffles in 14 (67%) experiments.

Ordinal regression with explainable distance metric learning based on ordered sequences

The purpose of this paper is to introduce a new distance metric learning algorithm for ordinal regression. Ordinal regression addresses the problem of predicting classes for which there is a natural ordering, but the real distances between classes are unknown. Since ordinal regression walks a fine line between standard regression and classification, it is a common pitfall to either apply a regression-like numerical treatment of variables or underrate the ordinal information applying nominal classification techniques. On a different note, distance metric learning is a discipline that has proven to be very useful when improving distance-based algorithms such as the nearest neighbors classifier. In addition, an appropriate distance can enhance the explainability of this model. In our study we propose an ordinal approach to learning a distance, called chain maximizing ordinal metric learning. It is based on the maximization of ordered sequences in local neighborhoods of the data. This approach takes into account all the ordinal information in the data without making use of any of the two extremes of classification or regression, and it is able to adapt to data for which the class separations are not clear. We also show how to extend the algorithm to learn in a non-linear setup using kernel functions. We have tested our algorithm on several ordinal regression problems, showing a high performance under the usual evaluation metrics in this domain. Results are verified through Bayesian non-parametric testing. Finally, we explore the capabilities of our algorithm in terms of explainability using the case-based reasoning approach. We show these capabilities empirically on two different datasets, experiencing significant improvements over the case-based reasoning with the traditional Euclidean nearest neighbors.

Tackling Ordinal Regression Problem for Heterogeneous Data: Sparse and Deep Multi-Task Learning Approaches.

Many real-world datasets are labeled with natural orders, i.e., ordinal labels. Ordinal regression is a method to predict ordinal labels that finds a wide range of applications in data-rich domains, such as natural, health and social sciences. Most existing ordinal regression approaches work well for independent and identically distributed (IID) instances via formulating a single ordinal regression task. However, for heterogeneous non-IID instances with well-defined local geometric structures, e.g., subpopulation groups, multi-task learning (MTL) provides a promising framework to encode task (subgroup) relatedness, bridge data from all tasks, and simultaneously learn multiple related tasks in efforts to improve generalization performance. Even though MTL methods have been extensively studied, there is barely existing work investigating MTL for heterogeneous data with ordinal labels. We tackle this important problem via sparse and deep multi-task approaches. Specifically, we develop a regularized multi-task ordinal regression (MTOR) model for smaller datasets and a deep neural networks based MTOR model for large-scale datasets. We evaluate the performance using three real-world healthcare datasets with applications to multi-stage disease progression diagnosis. Our experiments indicate that the proposed MTOR models markedly improve the prediction performance comparing with single-task ordinal regression models.

Convolutional Ordinal Regression Forest for Image Ordinal Estimation.

Image ordinal estimation is to predict the ordinal label of a given image, which can be categorized as an ordinal regression (OR) problem. Recent methods formulate an OR problem as a series of binary classification problems. Such methods cannot ensure that the global ordinal relationship is preserved since the relationships among different binary classifiers are neglected. We propose a novel OR approach, termed convolutional OR forest (CORF), for image ordinal estimation, which can integrate OR and differentiable decision trees with a convolutional neural network for obtaining precise and stable global ordinal relationships. The advantages of the proposed CORF are twofold. First, instead of learning a series of binary classifiers independently, the proposed method aims at learning an ordinal distribution for OR by optimizing those binary classifiers simultaneously. Second, the differentiable decision trees in the proposed CORF can be trained together with the ordinal distribution in an end-to-end manner. The effectiveness of the proposed CORF is verified on two image ordinal estimation tasks, i.e., facial age estimation and image esthetic assessment, showing significant improvements and better stability over the state-of-the-art OR methods.

Robust Ordinal Regression: User Credit Grading with Triplet Loss-Based Sampling

With the development of social media sites, user credit grading, which served as an important and fashionable problem, has attracted substantial attention from a slew of developers and operators of mobile applications. In particular, multi-grades of user credit aimed to achieve (1) anomaly detection and risk early warning and (2) personalized information and service recommendation for privileged users. The above two goals still remained as up-to-date challenges. To these ends, in this article, we propose a novel regression-based method. Technically speaking, we define three natural ordered categories including BlockList , GeneralList , and AllowList according to users’ registration and behavior information, which preserve both the global hierarchical relationship of user credit and the local coincident features of users, and hence formulate user credit grading as the ordinal regression problem. Our method is inspired by KDLOR ( kernel discriminant learning for ordinal regression ), which is an effective and efficient model to solve ordinal regression by mapping high-dimension samples to the discriminant region with supervised conditions. However, the performance of KDLOR is fragile to the extreme imbalanced distribution of users. To address this problem, we propose a robust sampling model to balance distribution and avoid overfit or underfit learning, which induces the triplet metric constraint to obtain hard negative samples that well represent the latent ordered class information. A step further, another salient problem lies in ambiguous samples that are noises or located in the classification boundary to impede optimized mapping and embedding. To this problem, we improve sampling by identifying and evading noises in triplets to obtain hard negative samples to enhance robustness and effectiveness for ordinal regression. We organized training and testing datasets for user credit grading by selecting limited items from real-life huge tables of users in the mobile application, which are used in similar problems; moreover, we theoretically and empirically demonstrate the advantages of the proposed model over established datasets.

On removing potential redundant constraints for SVOR learning

As an extension of support vector machine in ordinal regression problem, support vector ordinal regression (SVOR) finds (r−1) parallel hyperplanes by solving a quadratic programming which has n(r−1) constraints. Here, n and r represent the number of training sample and ranks, respectively. Therefore, it would costs much more time to train SVOR than SVC or SVR. Fortunately, the solution of SVOR is only decided by minor constraints which are associated with non-zero Lagrange multipliers. Other constraints with zero Lagrange multipliers have no influence on the solution. Because a training sample is associated with (r−1) constraints, retaining potential support vector may still induce that many redundant constraints are reserved. In this paper, we try to remove these potential redundant constraints for SVOR learning. For the jth parallel hyperplane, the potential constraints with non-zero Lagrange multipliers are associated with the samples near the jth parallel hyperplane. These samples can be identified by a chain near the jth parallel hyperplane. Then, other constraints for the jth parallel hyperplane can be discarded before learning. The number of the constraints can be reduced to less than 17 percent of the original in our experiments. Additionally, it only executes once to find potential critical constraints. Obviously, it is easy to tune parameters after removing potential redundant constraints. The experimental results on several datasets demonstrate that SVOR becomes much faster after removing potential redundant constraints and the performance does not degrade seriously.

Building Graduate Salary Grading Prediction Model Based on Deep Learning

Predicting salary trends of students after employment is vital for helping students to develop their career plans. Particularly, salary is not only considered employment information for students to pursue jobs, but also serves as an important indicator for measuring employability and competitiveness of graduates. This paper considers salary prediction as an ordinal regression problem and uses deep learning techniques to build a salary prediction model for determining the relative ordering between different salary grades. Specifically, to solve this problem, the model uses students’ personal information, grades, and family data as input features and employs a multi-output deep neural network to capture the correlation between the salary grades during training. Moreover, the model is pre-trained using a stacked denoising autoencoder and the corresponding weights after pre-training are used as the initial weights of the neural network. To improve the performance of model, dropout and bootstrap aggregation are used. The experimental results are very encouraging. With the predictive salary grades for graduates, school’s researchers can have a clear understanding of salary trends in order to promote student employability and competitiveness.

Non-parallel hyperplanes ordinal regression machine

This paper proposes a method to solve ordinal regression problems, namely the non-parallel hyperplanes ordinal regression machine (NPHORM). The goal of this approach is to find K different hyperplanes for the K classes with ordinal information, so that each class is as close as possible to the corresponding hyperplane while as far as possible from the adjacent to the left and right classes. The more flexible separate hyperplanes are preferred using the order information of the data. As a result, this approach only needs to solve K quadratic programming problems independently. Our approach NPHORM is validated on 2 artificial datasets, 16 discretized regression datasets and 17 real ordinal regression datasets and compared with 8 outstanding SVM-based ordinal regression approaches. The results show that our approach NPHORM is comparable with the other SVM-based approaches, especially in real ordinal regression datasets. In addition, our NPHORM is also carried out on the historical color image dataset to compare the performance of deep learning method. Experimental results demonstrate that the performance of our NPHORM outperforms the deep learning methods on MAE.

Height Estimation From Single Aerial Images Using a Deep Ordinal Regression Network

Understanding the 3D geometric structure of the Earth's surface has been an active research topic in photogrammetry and remote sensing community for decades, serving as an essential building block for various applications such as 3D digital city modeling, change detection, and city management. Previous researches have extensively studied the problem of height estimation from aerial images based on stereo or multi-view image matching. These methods require two or more images from different perspectives to reconstruct 3D coordinates with camera information provided. In this paper, we deal with the ambiguous and unsolved problem of height estimation from a single aerial image. Driven by the great success of deep learning, especially deep convolution neural networks (CNNs), some researches have proposed to estimate height information from a single aerial image by training a deep CNN model with large-scale annotated datasets. These methods treat height estimation as a regression problem and directly use an encoder-decoder network to regress the height values. In this paper, we proposed to divide height values into spacing-increasing intervals and transform the regression problem into an ordinal regression problem, using an ordinal loss for network training. To enable multi-scale feature extraction, we further incorporate an Atrous Spatial Pyramid Pooling (ASPP) module to extract features from multiple dilated convolution layers. After that, a post-processing technique is designed to transform the predicted height map of each patch into a seamless height map. Finally, we conduct extensive experiments on ISPRS Vaihingen and Potsdam datasets. Experimental results demonstrate significantly better performance of our method compared to the state-of-the-art methods.

Solving large-scale support vector ordinal regression with asynchronous parallel coordinate descent algorithms

Ordinal regression is one of the most influential tasks of supervised learning. Support vector ordinal regression (SVOR) is an appealing method to tackle ordinal regression problems. However, due to the complexity in the formulation of SVOR and the high cost of kernel computation, traditional SVOR solvers are inefficient for large-scale training. To address this problem, in this paper, we first highlight a special SVOR formulation whose thresholds are described implicitly, so that the dual formulation is concise to apply the state-of-the-art asynchronous parallel coordinate descent algorithm, such as AsyGCD. To further accelerate the training for SVOR, we propose two novel asynchronous parallel coordinate descent algorithms, called AsyACGD and AsyORGCD respectively. AsyACGD is an accelerated extension of AsyGCD using active set strategy. AsyORGCD is specifically designed for SVOR that it can keep the ordered thresholds when it is training so that it can obtain good performance with lower time. Experimental results on several large-scale ordinal regression datasets demonstrate the superiority of our proposed algorithms.

A Deep Learning Model for the Accurate and Reliable Classification of Disc Degeneration Based on MRI Data.

Although magnetic resonance imaging-based formalized grading schemes for intervertebral disc degeneration offer improved reproducibility compared with purely subjective ratings, their intrarater and interrater reliability are not nearly good enough to be able to detect small to medium effects in clinical longitudinal studies. The aim of this study thus was to develop a method that enables automatic and therefore reproducible and reliable evaluation of disc degeneration based on conventional clinical image data and Pfirrmann's grading scheme. We propose a classifier based on a deep convolutional neural network that we trained on a large, manually evaluated data set of 1599 patients (7948 intervertebral discs). To improve upon the status quo, we focused on the quality of the training data and performed extensive hyperparameter optimization. We assessed the potential benefits of optimizing loss functions beyond common cross-entropy loss, such as soft kappa loss, ordinal cross-entropy loss, or regression losses. We furthermore experimented with ways to mitigate class imbalance by pooling classes or using class-weighted loss functions. During model development and hyperparameter optimization, we used a fixed 90%/10% training/validation set split. To estimate real-world prediction performance, we performed 10-fold cross-validation. The evaluated image data results in a Gaussian degeneration grade distribution, and thus grades 1 and 5 are slightly underrepresented in the training set. Our default cross-entropy-based classifier achieves a reliability of κ = 0.92 (Cohen κ), an average sensitivity of 90.2%, and an average precision of 92.5%. In 99.2% of validation cases, the network's prediction deviates at most 1 Pfirrmann grades from the ground truth. Framed as an ordinal regression problem, the mean absolute error between the ground truth and the prediction is 0.08 Pfirrmann grade with a correlation of r = 0.96. The results of the 10-fold cross validation confirm those performance estimates, indicating no substantial overfitting. More sophisticated loss functions, class-based loss weighting, or class pooling did not lead to improved classification performance overall. With a reliability of κ > 0.9, our system clearly outperforms average human interrater as well as intrarater reliability. With an average sensitivity of more than 90%, our classifier also surpasses state-of-the-art machine learning solutions for automatically grading disc degeneration.

Quadruply Stochastic Gradient Method for Large Scale Nonlinear Semi-Supervised Ordinal Regression AUC Optimization

Semi-supervised ordinal regression (S2OR) problems are ubiquitous in real-world applications, where only a few ordered instances are labeled and massive instances remain unlabeled. Recent researches have shown that directly optimizing concordance index or AUC can impose a better ranking on the data than optimizing the traditional error rate in ordinal regression (OR) problems. In this paper, we propose an unbiased objective function for S2OR AUC optimization based on ordinal binary decomposition approach. Besides, to handle the large-scale kernelized learning problems, we propose a scalable algorithm called QS3ORAO using the doubly stochastic gradients (DSG) framework for functional optimization. Theoretically, we prove that our method can converge to the optimal solution at the rate of O(1/t), where t is the number of iterations for stochastic data sampling. Extensive experimental results on various benchmark and real-world datasets also demonstrate that our method is efficient and effective while retaining similar generalization performance.

Ordinal regression algorithms for the analysis of convective situations over Madrid-Barajas airport

Abstract In this paper we tackle a problem of convective situations analysis at Adolfo-Suarez Madrid-Barajas International Airport (Spain), based on Ordinal Regression algorithms. The diagnosis of convective clouds is key in a large airport like Barajas, since these meteorological events are associated with strong winds and local precipitation, which may affect air and land operations at the airport. In this work, we deal with a 12-h time horizon in the analysis of convective clouds, using as input variables data from a radiosonde station and also from numerical weather models. The information about the objective variable (convective clouds presence at the airport) has been obtained from the Madrid-Barajas METAR and SPECI aeronautical reports. We treat the problem as an ordinal regression task, where there exist a natural order among the classes. Moreover, the classification problem is highly imbalanced, since there are very few convective clouds events compared to clear days. Thus, a process of oversampling is applied to the database in order to obtain a better balance of the samples for this specific problem. An important number of ordinal regression methods are then tested in the experimental part of the work, showing that the best approach for this problem is the SVORIM algorithm, based on the Support Vector Machine strategy, but adapted for ordinal regression problems. The SVORIM algorithm shows a good accuracy in the case of thunderstorms and Cumulonimbus clouds, which represent a real hazard for the airport operations.

Data-driven Prognostics with Predictive Uncertainty Estimation using Ensemble of Deep Ordinal Regression Models

Prognostics or Remaining Useful Life (RUL) Estimation from multi-sensor time series data is useful to enable condition-based maintenance and ensure high operational availability of equipment. We propose a novel deep learning based approach for Prognostics with Uncertainty Quantification that is useful in scenarios where: (i) access to labeled failure data is scarce due to rarity of failures (ii) inherent noise ispresent in the sensor readings. The two scenarios mentioned are unavoidable sources of uncertainty in the RUL estimation process, often resulting in unreliable RUL estimates. To address (i), we formulate RUL estimation as an Ordinal Regression (OR) problem and propose LSTM-OR: deep Long Short Term Memory (LSTM) network-based approach to learn the OR function. We show that LSTM-OR naturally allows for the incorporation of censored operational instances in training along with the failed instances, leading to more robust learning. To address (ii), we propose a simple yet effectiveapproach to quantify predictive uncertainty in the RUL estimation models by training an ensemble of LSTM-OR models. Through empirical evaluation on the publicly available turbofan engine benchmark datasets, we demonstrate that LSTMOR is at par with commonly used deep metric regressionbased approaches for RUL estimation when sufficient failed instances are available for training. Importantly, LSTM-OR outperforms these metric regression-based approaches in the practical scenario where failed training instances are scarce, but sufficient operational (censored) instances are additionally available. Furthermore, our uncertainty quantification approach yields high-quality predictive uncertainty estimateswhile also leading to improved RUL estimates compared to single best LSTM-OR models.

An Algorithm for Ordinal Classification Based on Pairwise Comparison

Ordinal classification problems are applied in many fields. In the field of multivariate statistical analysis, these tasks are referred to as ordinal regression problems. In the field of management decision-making, they are known as multi-criteria decision analyses or sorting problems. This paper introduces the PairCode algorithm for ordinal classification with small sample sizes, which is based on a pairwise comparison strategy. In addition, this work outlines how to use pairwise comparisons to transform ordinal classifications into disordered regressions and how to transform the results of disordered regressions back to their original ordinal categories. Some effective strategies have been put forward, such as designing a class-label encoding matrix for the pairwise comparison, balancing samples, training classifiers, and predicting new samples. In numerical experiments, our algorithm (PairCode) is compared with the ordinal logistic regressions (LogisticOP) (Hu et al., IEEE Transactions on Knowledge and Data Engineering, 24(11), 2052–2064, 2012; Harrell 2015b), SVMOP (Gutierrez et al., IEEE Transactions on Knowledge and Data Engineering, 28(1), 127–146, 2016; Leathart et al. 2016), SVORIM (Chu and Sathiya Keerthi, Neural Computation, 19(3), 792–815 2007; Gutierrez et al., IEEE Transactions on Knowledge and Data Engineering, 28(1), 127–146, 2016), SVOREX (Chu and Sathiya Keerthi, Neural Computation, 19(3), 792–815 2007; Gutierrez et al., IEEE Transactions on Knowledge and Data Engineering, 28(1), 127–146, 2016), and ELMOP (Deng et al., Neurocomputing, 74(1), 447–456, 2010; Gutierrez et al., IEEE Transactions on Knowledge and Data Engineering, 28(1), 127–146, 2016). The results show that the PairCode algorithm performs better and is relatively stable as reflected by the correct classification rate (CCR), the mean absolute error (MAE), and the maximum MAE value (MMAE). However, the computing speed of the PairCode algorithm for classification is slightly slow and therefore warrants further study to improve the speed.