Mach Learn Res Research Articles

Achieving fairness with a simple ridge penalty

In this paper, we present a general framework for estimating regression models subject to a user-defined level of fairness. We enforce fairness as a model selection step in which we choose the value of a ridge penalty to control the effect of sensitive attributes. We then estimate the parameters of the model conditional on the chosen penalty value. Our proposal is mathematically simple, with a solution that is partly in closed form and produces estimates of the regression coefficients that are intuitive to interpret as a function of the level of fairness. Furthermore, it is easily extended to generalised linear models, kernelised regression models and other penalties, and it can accommodate multiple definitions of fairness. We compare our approach with the regression model from Komiyama et al. (in: Proceedings of machine learning research. 35th international conference on machine learning (ICML), vol 80, pp 2737–2746, 2018), which implements a provably optimal linear regression model and with the fair models from Zafar et al. (J Mach Learn Res 20:1–42, 2019). We evaluate these approaches empirically on six different data sets, and we find that our proposal provides better goodness of fit and better predictive accuracy for the same level of fairness. In addition, we highlight a source of bias in the original experimental evaluation in Komiyama et al. (in: Proceedings of machine learning research. 35th international conference on machine learning (ICML), vol 80, pp 2737–2746, 2018).

A communication-efficient method for ℓ0 regularization linear regression models

We propose a communication-efficient distributed learning algorithm for high-dimensional sparse linear regression models in the scenario that the data are stored across multiple machines. Our approach is a distributed version of the SDAR [Huang J, Jiao Y, Liu Y, et al. A constructive approach to l0 penalized regression. J Mach Learn Res. 2018;19(1):403–439] method for solving the KKT system of the regularized least squares. At each step of the proposed method, the reduced least squares are solved by the steepest descent method, which only needs to calculate the gradient vectors on each node machine and communicate them instead of the data. We refer to this as SD-SDAR for brevity. Under some regular conditions, we obtain the sharp and error bounds for the solution sequences generated by SD-SDAR algorithm. We investigate the computational complexity and show that the number of rounds of communications are bounded by and , respectively, where J is the number of important predictors, R is the relative magnitude of the non-zero target coefficients, N is the total sample size and p is the dimension of covariates. Simulation studies illustrate that SD-SDAR outperforms some existing distributed methods in accuracy, efficiency and support recovery.

Diagnostic Performance of a Noninvasive Breath Test for Colorectal Cancer: COBRA1 Study

Diagnostic Performance of a Noninvasive Breath Test for Colorectal Cancer: COBRA1 Study

Obstructive sleep apnea predicts 10-year cardiovascular disease-related mortality in the Sleep Heart Health Study: a machine learning approach.

Obstructive sleep apnea (OSA) is considered to be an important risk factor for the development of cardiovascular disease (CVD). This study aimed to develop and evaluate a machine learning approach with a set of features for assessing the 10-year CVD mortality risk of the OSA population. This study included 2,464 patients with OSA who met study inclusion criteria and were selected from the Sleep Heart Health Study. We evaluated the importance of potential features by mutual information. The top 9 features were selected to develop a random forest model. We evaluated the model performance on a test set (n = 493) using the area under the receiver operating curve with 95% confidence interval and confusion matrix. A random forest model awarded the highest area under the receiver operating curve of 0.84 (95% confidence interval: 0.78-0.89). The specificity and sensitivity were 73.94% and 81.82%, respectively. Sixty-three years old was a threshold for increased risk of 10-year CVD mortality. Persons with severe OSA had higher risk than those with mild OSA. This study demonstrated that a random forest model can provide a quick assessment of the risk of 10-year CVD mortality. Our model may be more informative for patients with OSA in determining their future CVD mortality risk. Li A, Roveda JM, Powers LS, Quan SF. Obstructive sleep apnea predicts 10-year cardiovascular disease-related mortality in the Sleep Heart Health Study: a machine learning approach. J Clin Sleep Med. 2022;18(2):497-504.

Extracellular vesicle-derived microbiome obtained from exhaled breath condensate in patients with asthma

Extracellular vesicle-derived microbiome obtained from exhaled breath condensate in patients with asthma

Concentration bounds for temporal difference learning with linear function approximation: the case of batch data and uniform sampling

We propose a stochastic approximation (SA) based method with randomization of samples for policy evaluation using the least squares temporal difference (LSTD) algorithm. Our proposed scheme is equivalent to running regular temporal difference learning with linear function approximation, albeit with samples picked uniformly from a given dataset. Our method results in an O(d) improvement in complexity in comparison to LSTD, where d is the dimension of the data. We provide non-asymptotic bounds for our proposed method, both in high probability and in expectation, under the assumption that the matrix underlying the LSTD solution is positive definite. The latter assumption can be easily satisfied for the pathwise LSTD variant proposed by Lazaric (J Mach Learn Res 13:3041–3074, 2012). Moreover, we also establish that using our method in place of LSTD does not impact the rate of convergence of the approximate value function to the true value function. These rate results coupled with the low computational complexity of our method make it attractive for implementation in big data settings, where d is large. A similar low-complexity alternative for least squares regression is well-known as the stochastic gradient descent (SGD) algorithm. We provide finite-time bounds for SGD. We demonstrate the practicality of our method as an efficient alternative for pathwise LSTD empirically by combining it with the least squares policy iteration algorithm in a traffic signal control application. We also conduct another set of experiments that combines the SA-based low-complexity variant for least squares regression with the LinUCB algorithm for contextual bandits, using the large scale news recommendation dataset from Yahoo.

Prediction models for COVID-19 clinical decision making

Prediction models for COVID-19 clinical decision making

Parallel generalized elliptical slice sampling with adaptive regional pseudo-priors

MCMC algorithm is well-known for having difficulty exploring distant modes when the target distribution is multi-modal. The reason is that a proposal state is likely to be rejected when traversing across low density regions. Focusing on this issue, we proposed parallel generalized elliptical slice sampling algorithm with adaptive regional pseudo-prior (RGESS). Different from the work of Fagan et al. [2016. Elliptical slice sampling with expectation propagation. In: UAI] and Nishihara et al. [Parallel mcmc with generalized elliptical slice sampling. J Mach Learn Res. 2014;15:2087–2112], different pseudo-priors are used at different regions to conduct the generalized elliptical slice sampling (GESS) algorithm. The rejection rate is modified to guarantee detailed balance condition. We also employ adaptive transition kernel and parallel computing to accelerate sampling speed. Experimental results on one synthetic and one real-world dataset show that the proposed algorithm has the following advantages: with the same starting points, the proposed algorithm spend less time to find the modes of the target distribution; after finding the modes, the proposed algorithm has less interactions between modes due to parameters adaption, which leads to lower rejection rate; when estimating the parameters of multi-modal posterior distributions, the samples generated by proposed algorithm can find different modes better.

Topic modeling, long texts and the best number of topics. Some Problems and solutions

The main aim of this article is to present the results of different experiments focused on the problem of model fitting process in topic modeling and its accuracy when applied to long texts. At the same time, in fact, the digital era has made available both enormous quantities of textual data and technological advances that have facilitated the development of techniques to automate the data coding and analysis processes. In the ambit of topic modeling, different procedures were born in order to analyze larger and larger collections of texts, namely corpora, but this has posed, and continues to pose, a series of methodological questions that urgently need to be resolved. Therefore, through a series of different experiments, this article is based on the following consideration: taking into account Latent Dirichlet Allocation (LDA), a generative probabilistic model (Blei et al. in J Mach Learn Res 3:993–1022, 2003; Blei and Lafferty in: Srivastava, Sahami (eds) Text mining: classification, clustering, and applications, Chapman & Hall/CRC Press, Cambridge, 2009; Griffiths and Steyvers in Proc Natl Acad Sci USA (PNAS), 101(Supplement 1):5228–5235, 2004), the problem of fitting model is crucial because the LDA algorithm demands that the number of topics is specified a priori. Needles to say, the number of topics to detect in a corpus is a parameter which affect the analysis results. Since there is a lack of experiments applied to long texts, our article tries to shed new light on the complex relationship between texts’ length and the optimal number of topics. In the conclusions, we present a clear-cut power-law relation between the optimal number of topics and the analyzed sample size, and we formulate it in a form of a mathematical model.

Independence test and canonical correlation analysis based on the alignment between kernel matrices for multivariate functional data

In the case of vector data, Gretton et al. (Algorithmic learning theory. Springer, Berlin, pp 63–77, 2005) defined Hilbert–Schmidt independence criterion, and next Cortes et al. (J Mach Learn Res 13:795–828, 2012) introduced concept of the centered kernel target alignment (KTA). In this paper we generalize these measures of dependence to the case of multivariate functional data. In addition, based on these measures between two kernel matrices (we use the Gaussian kernel), we constructed independence test and nonlinear canonical variables for multivariate functional data. We show that it is enough to work only on the coefficients of a series expansion of the underlying processes. In order to provide a comprehensive comparison, we conducted a set of experiments, testing effectiveness on two real examples and artificial data. Our experiments show that using functional variants of the proposed measures, we obtain much better results in recognizing nonlinear dependence.

Dirichlet Bayesian network scores and the maximum relative entropy principle

A classic approach for learning Bayesian networks from data is to identify a maximum a posteriori (MAP) network structure. In the case of discrete Bayesian networks, MAP networks are selected by maximising one of several possible Bayesian–Dirichlet (BD) scores; the most famous is the Bayesian–Dirichlet equivalent uniform (BDeu) score from Heckerman et al. (Mach Learn 20(3):197–243, 1995). The key properties of BDeu arise from its uniform prior over the parameters of each local distribution in the network, which makes structure learning computationally efficient; it does not require the elicitation of prior knowledge from experts; and it satisfies score equivalence. In this paper we will review the derivation and the properties of BD scores, and of BDeu in particular, and we will link them to the corresponding entropy estimates to study them from an information theoretic perspective. To this end, we will work in the context of the foundational work of Giffin and Caticha (Proceedings of the 27th international workshop on Bayesian inference and maximum entropy methods in science and engineering, pp 74–84, 2007), who showed that Bayesian inference can be framed as a particular case of the maximum relative entropy principle. We will use this connection to show that BDeu should not be used for structure learning from sparse data, since it violates the maximum relative entropy principle; and that it is also problematic from a more classic Bayesian model selection perspective, because it produces Bayes factors that are sensitive to the value of its only hyperparameter. Using a large simulation study, we found in our previous work [Scutari in J Mach Learn Res (Proc Track PGM 2016) 52:438–448, 2016] that the Bayesian–Dirichlet sparse (BDs) score seems to provide better accuracy in structure learning; in this paper we further show that BDs does not suffer from the issues above, and we recommend to use it for sparse data instead of BDeu. Finally, will show that these issues are in fact different aspects of the same problem and a consequence of the distributional assumptions of the prior.

Real-time image recognition using weighted spatial pyramid networks

The latest-generation earth observation instruments on airborne and satellite platforms are currently producing an almost continuous high-dimensional data stream. This exponentially growing data poses a new challenge for real-time image processing and recognition. Making full and effective use of the spectral information and spatial structure information of high-resolution remote sensing image is the key to the processing and recognition of high-resolution remote sensing data. In this paper, the adaptive multipoint moment estimation (AMME) stochastic optimization algorithm is proposed for the first time by using the finite lower-order moments and adding the estimating points. This algorithm not only reduces the probability of local optimum in the learning process, but also improves the convergence rate of the convolutional neural network (Lee Cun et al. in Advances in neural information processing systems, 1990). Second, according to the remote sensing image with characteristics of complex background and small sensitive targets, and by automatic discovery, locating small targets, and giving high weights, we proposed a feature extraction method named weighted pooling to further improve the performance of real-time image recognition. We combine the AMME and weighted pooling with the spatial pyramid representation (Harada et al. in Comput Vis Pattern Recognit 1617–1624, 2011) algorithm to form a new, multiscale, and multilevel real-time image recognition model and name it weighted spatial pyramid networks (WspNet). At the end, we use the MNIST, ImageNet, and natural disasters under remote sensing data sets to test WspNet. Compared with other real-time image recognition models, WspNet achieve a new state of the art in terms of convergence rate and image feature extraction compared with conventional stochastic gradient descent method [like AdaGrad, AdaDelta and Adam (Zeiler in Comput Sci, 2012; Kingma and Ba in Comput Sci, 2014; Duchi et al. in J Mach Learn Res 12(7):2121–2159, 2011] and pooling method [like max-pooling, avg-pooling and stochastic-pooling (Zeiler and Fergus in stochastic-pooling for regularization of deep convolutional neural networks, 2013)].

High-probability minimax probability machines

In this paper we focus on constructing binary classifiers that are built on the premise of minimising an upper bound on their future misclassification rate. We pay particular attention to the approach taken by the minimax probability machine (Lanckriet et al. in J Mach Learn Res 3:555–582, 2003), which directly minimises an upper bound on the future misclassification rate in a worst-case setting: that is, under all possible choices of class-conditional distributions with a given mean and covariance matrix. The validity of these bounds rests on the assumption that the means and covariance matrices are known in advance, however this is not always the case in practice and their empirical counterparts have to be used instead. This can result in erroneous upper bounds on the future misclassification rate and lead to the formulation of sub-optimal predictors. In this paper we address this oversight and study the influence that uncertainty in the moments, the mean and covariance matrix, has on the construction of predictors under the minimax principle. By using high-probability upper bounds on the deviation between true moments and their empirical counterparts, we can re-formulate the minimax optimisation to incorporate this uncertainty and find the predictor that minimises the high-probability, worst-case misclassification rate. The moment uncertainty introduces a natural regularisation component into the optimisation, where each class is regularised in proportion to the degree of moment uncertainty. Experimental results would support the view that in the case of with limited data availability, the incorporation of moment uncertainty can lead to the formation of better predictors.

Comparison of a massive and diverse collection of ensembles and other classifiers for oil spill detection in SAR satellite images

We present a comparison of the largest collection of classifiers considered until now in the literature, composed by 428 methods belonging to 41 very different families. This collection, much larger than the one in our previous work (Fernandez-Delgado et al. in J Mach Learn Res 15:3133–3181, 2014), includes 320 ensembles (varying the base and meta-classifiers), alongside with Support Vector Machines, Bayesian, Neural Networks, Discriminant Analysis, Multivariate Adaptive Regression Splines, Random Forests, Decision Trees and many others. The classifier comparison is developed on the detection of oil spills on Synthetic Aperture Radar (SAR) images taken from satellites. The SAR images have revealed very useful to surveillance maritime agencies for the detection of regular offshore operational discharges, which, despite is commonly accepted, is one of the biggest causes of hydrocarbon marine pollution, instead of tanker and oil platform catastrophes. After a segmentation of the SAR images to select oil spill candidates, classifiers use the features extracted from these candidates to discard frequent and expensives look-alikes (false positives), caused by natural phenomena. Testing experiments revealed that the RotationForest ensemble of MultilayerPerceptron base classifiers, applying Kernel PCA on the original data, achieves the best accuracy and Cohen \(\kappa\) (87.1 % and 71.0 %, respectively) with a low frequency of false positives (5.13 %).

Estimating FLEimage distributions of manual fiducial localization in CT images.

PurposeThe fiducial localization error distribution (FLE) and fiducial configuration govern the application accuracy of point-based registration and drive target registration error (TRE) prediction models. The error of physically localizing patient fiducials ({hbox {FLE}}_mathrm{patient}) is negligible when a registration probe matches the implanted screws with mechanical precision. Reliable trackers provide an unbiased estimate of the positional error ({hbox {FLE}}_mathrm{tracker}) with cheap repetitions. FLE further contains the localization error in the imaging data ({hbox {FLE}}_mathrm{image}), sampling of which in general is expensive and possibly biased. Finding the best techniques for estimating {hbox {FLE}}_mathrm{image} is crucial for the applicability of the TRE prediction methods.MethodsWe built a ground-truth (gt)-based unbiased estimator (widehat{{hbox {FLE}}_mathrm{gt}}) of {hbox {FLE}}_mathrm{image} from the samples collected in a virtual CT dataset in which the true locations of image fiducials are known by definition. Replacing true locations in {hbox {FLE}}_mathrm{gt} by the sample mean creates a practical difference-to-mean (dtm)-based estimator (widehat{{hbox {FLE}}_mathrm{dtm}}) that is applicable on any dataset. To check the practical validity of the dtm estimator, ten persons manually localized nine fiducials ten times in the virtual CT and the resulting {hbox {FLE}}_mathrm{dtm} and {hbox {FLE}}_mathrm{gt} distributions were tested for statistical equality with a kernel-based two-sample test using the maximum mean discrepancy (MMD) (Gretton in J Mach Learn Res 13:723–773, 2012) statistics at alpha =0.05.Results{hbox {FLE}}_mathrm{dtm} and {hbox {FLE}}_mathrm{gt} were found (for most of the cases) not to be statistically significantly different; conditioning them on persons and/or screws however yielded statistically significant differences much more often.ConclusionsWe conclude that widehat{{hbox {FLE}}_mathrm{dtm}} is the best candidate (within our model) for estimating {hbox {FLE}}_mathrm{image} in homogeneous TRE prediction models. The presented approach also allows ground-truth-based numerical validation of {hbox {FLE}}_mathrm{image} estimators and (manual/automatic) image fiducial localization methods in phantoms with parameters similar to clinical datasets.

A rank-corrected procedure for matrix completion with fixed basis coefficients

For the problems of low-rank matrix completion, the efficiency of the widely-used nuclear norm technique may be challenged under many circumstances, especially when certain basis coefficients are fixed, for example, the low-rank correlation matrix completion in various fields such as the financial market and the low-rank density matrix completion from the quantum state tomography. To seek a solution of high recovery quality beyond the reach of the nuclear norm, in this paper, we propose a rank-corrected procedure using a nuclear semi-norm to generate a new estimator. For this new estimator, we establish a non-asymptotic recovery error bound. More importantly, we quantify the reduction of the recovery error bound for this rank-corrected procedure. Compared with the one obtained for the nuclear norm penalized least squares estimator, this reduction can be substantial (around 50 %). We also provide necessary and sufficient conditions for rank consistency in the sense of Bach (J Mach Learn Res 9:1019---1048, 2008). Very interestingly, these conditions are highly related to the concept of constraint nondegeneracy in matrix optimization. As a byproduct, our results provide a theoretical foundation for the majorized penalty method of Gao and Sun (A majorized penalty approach for calibrating rank constrained correlation matrix problems. http://www.math.nus.edu.sg/~matsundf/MajorPen_May5.pdf, 2010) and Gao (2010) for structured low-rank matrix optimization problems. Extensive numerical experiments demonstrate that our proposed rank-corrected procedure can simultaneously achieve a high recovery accuracy and capture the low-rank structure.

Coordinate Descent Algorithm for Ramp Loss Linear Programming Support Vector Machines

In order to control the effects of outliers in training data and get sparse results, Huang et al. (J Mach Learn Res 15:2185---2211, 2014) proposed the ramp loss linear programming support vector machine. This combination of $$\mathrm {l}_1$$l1 regularization and ramp loss does not only lead to the sparsity of parameters in decision functions, but also limits the effects of outliers with a maximal penalty. However, due to its non-convexity, the computational cost to achieve a satisfying solution is often expensive. In this paper, we propose a modified coordinate descent algorithm, which deals with a series of one-variable piecewise linear subproblems. Considering that the obtained subproblems are DC programming problems, we linearize the concave part of the objective functions and solve the obtained convex problems. To test the performances of the proposed algorithm, numerical experiments have been carried out and analysed on benchmark data sets. To enhance the sparsity and robustness, the experiments are initialized from C-SVM solutions. The results confirm its excellent performances in classification accuracy, robustness and efficiency in computation.

Probabilistic characterization of nearest neighbor classifier

The k-nearest neighbor classification algorithm (kNN) is one of the most simple yet effective classification algorithms in use. It finds major applications in text categorization, outlier detection, handwritten character recognition, fraud detection and in other related areas. Though sound theoretical results exist regarding convergence of the generalization error (GE) of this algorithm to Bayes error, these results are asymptotic in nature. The understanding of the behavior of the kNN algorithm in real world scenarios is limited. In this paper, assuming categorical attributes, we provide a principled way of studying the non-asymptotic behavior of the kNN algorithm. In particular, we derive exact closed form expressions for the moments of the GE for this algorithm. The expressions are functions of the sample, and hence can be computed given any joint probability distribution defined over the input–output space. These expressions can be used as a tool that aids in unveiling the statistical behavior of the algorithm in settings of interest viz. an acceptable value of k for a given sample size and distribution. Moreover, Monte Carlo approximations of such closed form expressions have been shown in Dhurandhar and Dobra (J Mach Learn Res 9, 2008; ACM Trans Knowl Discov Data 3, 2009) to be a superior alternative in terms of speed and accuracy when compared with computing the moments directly using Monte Carlo. This work employs the semi-analytical methodology that was proposed recently to better understand the non-asymptotic behavior of learning algorithms.

One-dimensional center-based l 1-clustering method

Motivated by the method for solving center-based Least Squares—clustering problem (Kogan in Introduction to clustering large and high-dimensional data, Cambridge University Press, 2007; Teboulle in J Mach Learn Res 8:65–102, 2007) we construct a very efficient iterative process for solving a one-dimensional center-based l1—clustering problem, on the basis of which it is possible to determine the optimal partition. We analyze the basic properties and convergence of our iterative process, which converges to a stationary point of the corresponding objective function for each choice of the initial approximation. Given is also a corresponding algorithm, which in only few steps gives a stationary point and the corresponding partition. The method is illustrated and visualized on the example of looking for an optimal partition with two clusters, where we check all stationary points of the corresponding minimizing functional. Also, the method is tested on the basis of large numbers of data points and clusters and compared with the method for solving the center-based Least Squares—clustering problem described in Kogan (2007) and Teboulle (2007).

Classifying cognitive states of brain activity via one-class neural networks with feature selection by genetic algorithms

It is generally assumed that one-class machine learning techniques can not reach the performance level of two-class techniques. The importance of this work is that while one-class is often the appropriate classification setting for identifying cognitive brain functions, most work in the literature has focused on two-class methods. In this paper, we demonstrate how one-class recognition of cognitive brain functions across multiple subjects can be performed at the 90% level of accuracy via an appropriate choice of features which can be chosen automatically. Our work extends one-class work by Hardoon and Manevitz (fMRI analysis via one-class machine learning techniques. In: Proceedings of the Nineteenth IJCAI, pp 1604–1605, 2005), where such classification was first shown to be possible in principle albeit with an accuracy of about 60%. The results of this paper are also comparable to work of various groups around the world e.g. Cox and Savoy (NeuroImage 19:261–270, 2003), Mourao-Miranda et al. (NeuroImage, 2006) and Mitchell et al., (Mach Learn 57:145–175, 2004) which have concentrated on two-class classification. The strengthening in the feature selection was accomplished by the use of a genetic algorithm run inside the context of a wrapper approach around a compression neural network for the basic one-class identification. In addition, versions of one-class SVM due to Scholkopf et al. (Estimating the support of a high-dimensional distribution. Technical Report MSR-TR-99-87, Microsoft Research, 1999) and Manevitz and Yousef (J Mach Learn Res 2:139–154, 2001) were investigated.