Non-parametric Statistical Learning Research Articles

Mortgage Loan Data Exploration with Non-parametric Statistical and Machine Learning Perspectives

Mortgage Loan Data Exploration with Non-parametric Statistical and Machine Learning Perspectives

Functional clustering of NPLs recovery curves

The recovery performance of a portfolio of Non-Performing Loans can be measured in terms of recovery rate and liquidation time jointly through a “recovery curve” representative of recovery rates over time. When portfolio heterogeneity is very high, it is informative to estimate more than just one curve by dividing the portfolio into several homogeneous subsets, i.e. clusters, and calculating a recovery curve for each of them. The aim of this work is to estimate the optimal portfolio partition and the smoothed recovery curves of each cluster by means of non-parametric statistical learning techniques.

Distribution-free algorithms for predictive stochastic programming in the presence of streaming data

This paper studies a fusion of concepts from stochastic programming and non-parametric statistical learning in which data is available in the form of covariates interpreted as predictors and responses. Such models are designed to impart greater agility, allowing decisions under uncertainty to adapt to the knowledge of predictors (leading indicators). This paper studies two classes of methods for such joint prediction-optimization models. One of the methods may be classified as a first-order method, whereas the other studies piecewise linear approximations. Both of these methods are based on coupling non-parametric estimation for predictive purposes, and optimization for decision-making within one unified framework. In addition, our study incorporates several non-parametric estimation schemes, including k nearest neighbors (kNN) and other standard kernel estimators. Our computational results demonstrate that the new algorithms proposed in this paper outperform traditional approaches which were not designed for streaming data applications requiring simultaneous estimation and optimization as important design features for such algorithms. For instance, coupling kNN with Stochastic Decomposition (SD) turns out to be over 40 times faster than an online version of Benders Decomposition while finding decisions of similar quality. Such computational results motivate a paradigm shift in optimization algorithms that are intended for modern streaming applications.

A Framework for Utilizing Permutational Multiple Analysis of Variance as a Precursor for Nonparametric Statistical Learning with Cyber Network Data

INTRODUCTION: Although scientific hypothesis testing methodologies are well established, their application to falsifiable hypothesis testing for assessing causal relationships potentially identified by machine learning and artificial intelligence models is rare due to the primarily nonparametric statistical nature of these systems.OBJECTIVES: The primary objective of this study is to demonstrate the potential for applying nonparametric statistical tests to a mixed qualitative and quantitative cyber network dataset as a method to pre-assess the feasibility of applying forms of statistical hypothesis testing before a machine learning algorithm models the data.METHODS: A mixture of permuted analysis of variance models augmented by the use of transformed non-Euclidean multivariate distances between curated dependent variable classes produced this research data. Quasi-experimental data from an enclosed laboratory environment utilizing a monitored, locally unrestricted network that introduced known Internet of Things (IoT) malware software supplied network flow events.RESULTS: A PERMANOVA model was executed against 62,000 records of the network flow observations, using Euclidean distance measurements with variable-dependent relationship ordering, using terms added sequentially (first to last) in the order encountered in the raw network flow dataset, using 200 permutations. This precursor test resulted in a p-value for the PERMANOVA model that incorporated terms added sequentially of 0.02985, providing an F value of 0.00017 with which to determine the ratio of explained to unexplained variance. Utilizing an analysis of the F values for all of the residuals, we show 29,998 degrees of freedom with a residual F model score of 0.99983, indicating that there is a strong proportion of explained to unexplained variance across all of the independent variables contained in the model. The model is thus statistically significant with a p-value below the alpha test statistic of 0.05.CONCLUSION: This research has demonstrated that it is possible to apply tests of falsifiability that incorporate reproducible methods into the quasi-experiment design and apply this to the field of machine learning. Applied to AI/ML (artificial intelligence/machine learning) models, this pre-assessment methodology supports the appropriateness of cyber network flow datasets in which a final test of statistical significance would be required. The authors believe that this represents a substantially useful precursor assessment stage for the suitability and reliability of the utilization of any nonparametric statistical learning algorithms applied to cyber network data predictive analytics.

Measuring the Recovery Performance of a Portfolio of NPLs

The objective of the present paper is to propose a new method to measure the recovery performance of a portfolio of non-performing loans (NPLs) in terms of recovery rate and time to liquidate. The fundamental idea is to draw a curve representing the recovery rates over time, here assumed discretized, for example, in years. In this way, the user can get simultaneously information about recovery rate and time to liquidate of the portfolio. In particular, it is discussed how to estimate such a curve in the presence of right-censored data, e.g., when the NPLs composing the portfolio have been observed in different time periods, with a method based on an algorithm that is usually used in the construction of survival curves. The curves obtained are smoothed with nonparametric statistical learning techniques. The effectiveness of the proposal is shown by applying the method to simulated and real financial data. The latter are about some portfolios of Italian unsecured NPLs taken over by a specialized operator.

Fully automated analysis of NMR data with Bayesian nonparametric statistical learning.

The characterization of individual peaks in NMR spectra is an important but challenging step in the investigation of complex biomolecular systems. The major challenge arises due to peak identification which, conventionally, occurs prior to peak characterization. Unluckily, the identification of uncharacterized peaks is highly problematic because of noise and, most importantly, substantial overlap with nearby peaks. For this reason, often the analysis of NMR spectra proceeds by first relying on experts’ opinion to identify peaks and subsequent parameter fitting. Mathematically, peak identification is a problem of model selection and, within a nonparametric analysis framework, can be avoided altogether, this way eliminating the need for experts’ input. In this study, we apply Bayesian nonparametric statistics to develop a comprehensive mathematical framework for automated peak identification and characterization. Our approach relies on advanced statistical representations of FIDs and allows for: simultaneous identification and characterization of individual peaks in an NMR spectrum, accurate noise deconvolution, and extensions to multidimensional NMR or incorporation of specialized protocols. We show that our methods can correctly identify and characterize overlapping peaks without user input. We benchmark our analyses against ground truth peak characteristics using synthetic as well as experimental datasets.

Nonparametric statistical learning based on modal regression

In this article, we propose a novel nonparametric statistical learning tool based on modal regression, which can complement the standard mean and quantile regression and has broad applicability in various fields. We first propose a local polynomial modal regression which focuses on the most likely conditional value (conditional mode) of a dependent variable Y given covariates x, and has several superiorities over conditional mean or quantiles, such as resistance to outliers and some forms of measurement error and having shorter prediction intervals when data are skewed. We employ the idea of local polynomial regression to estimate the modal regression nonparametrically. To broaden the applicability of the new technique to multivariate data or functional/longitudinal data, we further develop a varying coefficient modal regression. A Monte Carlo simulation study and an analysis of health care expenditure data demonstrate that the new proposed nonparametric modal regression is a promising complementary nonparametric data analysis tool to conventional nonparametric mean or quantile regression.

Generational differences in automobility: Comparing America's Millennials and Gen Xers using gradient boosting decision trees

Whether the Millennials are less auto-centric than the previous generations has been widely discussed in the literature. Most existing studies use regression models and assume that all factors are linear-additive in contributing to the young adults' driving behaviors. This study relaxes this assumption by applying a non-parametric statistical learning method, namely the gradient boosting decision trees (GBDT). Using U.S. nationwide travel surveys for 2001 and 2017, this study examines the non-linear dose-response effects of lifecycle, socio-demographic and residential factors on daily driving distances of Millennial and Gen-X young adults. Holding all other factors constant, Millennial young adults had shorter predicted daily driving distances than their Gen-X counterparts. Besides, residential and economic factors explain around 50% of young adults' daily driving distances, while the collective contributions for life course events and demographics are about 33%. This study also identifies the density ranges for formulating effective land use policies aiming at reducing automobile travel demand.

Spinal Muscular Atrophy after Nusinersen Therapy: Improved Physiology in Pediatric Patients with No Significant Change in Urine, Serum, and Liquor 1H-NMR Metabolomes in Comparison to an Age-Matched, Healthy Cohort.

Spinal muscular atrophy (SMA) is a genetically heterogeneous group of rare neuromuscular diseases and was until recently the most common genetic cause of death in children. The effects of 2-month nusinersen therapy on urine, serum, and liquor 1H-NMR metabolomes in SMA males and females were not explored yet, especially not in comparison to the urine 1H-NMR metabolomes of matching male and female cohorts. In this prospective, single-centered study, urine, serum, and liquor samples were collected from 25 male and female pediatric patients with SMA before and after 2 months of nusinersen therapy and urine samples from a matching healthy cohort (n = 125). Nusinersen intrathecal application was the first therapy for the treatment of SMA by the Food and Drug Administration (FDA) and the European Medicines Agency (EMA). Metabolomes were analyzed using targeted metabolomics utilizing 600 MHz 1H-NMR, parametric and nonparametric multivariate statistical analyses, machine learning, and modeling. Medical assessment before and after nusinersen therapy showed significant improvements of movement, posture, and strength according to various medical tests. No significant differences were found in metabolomes before and after nusinersen therapy in urine, serum, and liquor samples using an ensemble of statistical and machine learning approaches. In comparison to a healthy cohort, 1H-NMR metabolomes of SMA patients contained a reduced number and concentration of urine metabolites and differed significantly between males and females as well. Significantly larger data scatter was observed for SMA patients in comparison to matched healthy controls. Machine learning confirmed urinary creatinine as the most significant, distinguishing SMA patients from the healthy cohort. The positive effects of nusinersen therapy clearly preceded or took place devoid of significant rearrangements in the 1H-NMR metabolomic makeup of serum, urine, and liquor. Urine creatinine was successful at distinguishing SMA patients from the matched healthy cohort, which is a simple systemic novelty linking creatinine and SMA to the physiology of inactivity and diabetes, and it facilitates the monitoring of SMA disease in pediatric patients through non-invasive urine collection.

A Hybrid Predictive Approach for Chromium Layer Thickness in the Hard Chromium Plating Process Based on the Differential Evolution/Gradient Boosted Regression Tree Methodology

The purpose of the industrial process of chromium plating is the creation of a hard and wear-resistant layer of chromium over a metallic surface. One of the main properties of chromium plating is its resistance to both wear and corrosion. This research presents an innovative nonparametric machine learning approach that makes use of a hybrid gradient boosted regression tree (GBRT) methodology for hard chromium layer thickness prediction. GBRT is a non-parametric statistical learning technique that produces a prediction model in the form of an ensemble of weak prediction models. The motivation for boosting is a procedure that combines the output of many weak classifiers to produce a powerful committee. In this study, the GBRT hyperparameters were optimized with the help of differential evolution (DE). DE is an optimization technique within evolutionary computing. The results found that this model was able to predict the thickness of the chromium layer formed in this industrial process with a determination coefficient equal to 0.9842 and a root-mean-square error value of 0.01590. The two most important variables of the model were the time of the hard-chromium process and the thickness of the layer removed by electropolishing. Thus, these results provide a foundation for an accurate predictive model of hard chromium layer thickness. The derived model also allowed the ranking of the importance of the independent input variables that were examined. Finally, the high performance and simplicity of the model make the DE/GBRT method attractive compared to conventional forecasting techniques.

Real-Time Impedance Control Based on Learned Inverse Dynamics

Impedance control of robotic manipulators allows them to perform interaction task where safety is the foremost requirement. A major hurdle in the implementation of impedance control has always been the requirement of availability and online real-time computation of dynamic model. This paper presents a complete data-driven, machine learning approach to impedance control in real time of an industrial manipulator. The technique used here to learn the inverse dynamic model of an industrial robot is based on an incremental nonparametric statistical learning approach and is called locally weighted projection regression. Unlike traditional model-based control schemes, the proposed control strategy requires less a priori knowledge of the system being modeled, is computationally efficient to run in hard real time, and can be updated during online operation of the arm. The main contribution of this paper is the development of learning-based impedance control scheme and its implementation in hard real time (at 500 Hz control loop rate) on an industrial robot (Barrett WAM arm). To validate performance of the proposed scheme, its comparison with a controller based on an analytically derived model (computed online in a numeric recursive way using Newton–Euler approach) has also been presented through experimentation in a laboratory environment.

Predictive inference with random forests: A new perspective on classical analyses

Despite the number of problems that can occur when core model assumptions are violated, nearly all quantitative political science research relies on inflexible regression models that require a linear relationship between dependent and independent variables for valid inference. We argue that nonparametric statistical learning methods like random forests are capable of combining the benefits of interpretability and flexibility. Recent work has shown that under suitable regularity conditions, averaging over predictions made by subsampled random forests produces asymptotically normal predictions. After estimating the variance, this property can be exploited to produce hypothesis tests and confidence intervals analogous to those produced within a parametric framework. We demonstrated the utility of this approach by replicating an important study on the determinants of civil war onset and show that subtle nonlinear relationships are uncovered, providing a new perspective on these ongoing research questions.

Case-only trees and random forests for exploring genotype-specific treatment effects in randomized clinical trials with dichotomous endpoints.

Discovering gene-treatment interactions in clinical trials is of rising interest in the era of precision medicine. Nonparametric statistical learning methods such as trees and random forests are useful tools for building prediction rules. In this article, we introduce trees and random forests to the recently proposed case-only approach for discovering gene-treatment interactions and estimating marker-specific treatment effects for a dichotomous trial endpoints. The motivational example is a case-control genetic association study in the Prostate Cancer Prevention Trial (PCPT), which tested the hypothesis whether finasteride can prevent prostate cancer. We compare this novel approach to the interaction tree method previously proposed. Because of the modeling simplicity - directly targeting at interaction - and the statistical efficiency of the case-only approach, case-only trees and random forests yield more accurate prediction of heterogeneous treatment effects and better measure of variable importance, relative to the interaction tree method which uses data from both cases and controls. Application of the proposed case-only trees and random forests to the PCPT study yielded a discovery of genotypes that may influence the prevention effect of finasteride.

Nonparametric inference of interaction laws in systems of agents from trajectory data.

Inferring the laws of interaction in agent-based systems from observational data is a fundamental challenge in a wide variety of disciplines. We propose a nonparametric statistical learning approach for distance-based interactions, with no reference or assumption on their analytical form, given data consisting of sampled trajectories of interacting agents. We demonstrate the effectiveness of our estimators both by providing theoretical guarantees that avoid the curse of dimensionality and by testing them on a variety of prototypical systems used in various disciplines. These systems include homogeneous and heterogeneous agent systems, ranging from particle systems in fundamental physics to agent-based systems that model opinion dynamics under the social influence, prey-predator dynamics, flocking and swarming, and phototaxis in cell dynamics.

Spatio-statistical optimization of image segmentation process for building footprint extraction using very high-resolution WorldView 3 satellite data

Segmentation process in building footprint extraction using object-based image analysis is crucial due to several factors, such as the spatial and spectral resolution of remote sensing images and the complexity of geo-objects. Consequently, the selection of suitable parameters to ensure the best segmentation quality remains a challenge. To overcome this issue, a spatio-statistical optimization technique that combines the Taguchi statistical method and a spatial plateau objective function (POF) was developed to extract building footprint from high-resolution Worldview 3 (WV3) satellite data. Initially, the Taguchi statistical method was used to design the orthogonal array of 25 experiments with three segmentation parameters, namely, scale, shape, and compactness, each having five varying values that directly affect the quality of segmentation. Asserted that, the scale factor was classified into small and large scales to avoid over-segmentation and under-segmentation problems. Afterwards, the POF, which is also a spatial optimization approach for evaluating segmentation quality, was computed for each experiment using their respective level combinations. Next, the combination of factor level in the orthogonal array and the calculated POF was merged to produce main effects and interaction plots for signal-to-noise ratios (SNR), whereby the smaller-is-better and larger-is-better options of the Taguchi’s SNR were tested on each parameter to maximize their effects. After that, the segmentation quality obtained from the proposed method was assessed by comparing with the benchmark method introduced by Dragut and result indicates that the proposed method was better than the benchmark method. Subsequently, the final optimal parameters were used for segmentation process in eCognition and the image object was classified into five land cover classes (building, road, water, trees, and grass) by using a supervised non-parametric statistical learning technique, support vector machine classifier. Finally, the building features was extracted, and the detection accuracy was evaluated based on receiver operating characteristics (ROC). Result shows the area under ROC curve (AUC) of 0.804 with p < 0.0001 at 95% confidence level. This verifies that the proposed method is effective for building detection with high accuracy and the integration of Taguchi and objective function managed to determine the optimal segmentation parameters. Optimization segmentation parameters can later be applied to distinguish roof materials and conditions.

A Machine Learning Metasystem for Robust Probabilistic Nonlinear Regression-Based Forecasting of Seasonal Water Availability in the US West

Hydroelectric power generation, water supplies for municipal, agricultural, manufacturing, and service industry uses including technology-sector requirements, dam safety, flood control, recreational uses, and ecological and legal constraints, all place simultaneous, competing demands on the heavily stressed water management infrastructure of the mostly arid American West. Optimally managing these resources depends on predicting water availability. We built a probabilistic nonlinear regression water supply forecast (WSF) technique for the US Department of Agriculture, which runs the largest stand-alone WSF system in the US West. Design criteria included improved accuracy over the existing system; uncertainty estimates that seamlessly handle complex (heteroscedastic, non-Gaussian) prediction errors; integration of physical hydrometeorological process knowledge and domain-specific expert experience; ability to accommodate nonlinearity, model selection uncertainty and equifinality, and predictor multicollinearity and high dimensionality; and relatively easy, low-cost implementation. Some methods satisfied some of these requirements but none met all, leading us to develop a novel, interdisciplinary, and pragmatic prediction metasystem through a carefully considered synthesis of well-established, off-the-shelf components and approaches, spanning supervised and unsupervised machine learning, nonparametric statistical modeling, ensemble learning, and evolutionary optimization, focusing on maintaining but radically updating the principal components regression framework widely used for WSF. Testing this integrated multi-method prediction engine demonstrated its value for river forecasting; USDA adoption is a landmark for transitioning machine learning from research into practice in this field. Its ability to handle all the foregoing design criteria and requirements, which are not unique to WSF, suggests potential for extension to complex probabilistic prediction problems in other fields.

A Predictive Modeling for Detecting Fraudulent Automobile Insurance Claims

Fraudulent automobile insurance claims are not only a loss for insurance companies, but also for their policyholders. The goal of this research is to develop, first, a decision-making algorithm to classify whether a claim is classified as fraudulent or not; and, second, what types of variables should be focused to detect fraudulent claims. To achieve this goal, highly accurate prediction models are built by discovering important sets of features via variable selection algorithms, which can in turn help prevent future loss. In this research, parametric and nonparametric statistical learning algorithms are considered to reduce uncertainty and increase the chances of detecting the appropriate claims. An important set of features for a model is determined by measuring variable importance based on the observed characteristics of a claim via a cross-validation and by testing improvement of the performance at which automobile fraudulent claims are accurately classified using Akaike Information Criterion. We could achieve accuracy above 95% with a set of features selected via a cross-validation. This research would offer some benefit to the insurance industry for their fraud detection research in order to prevent insurance abuse from escalating any further.

APPLICATION OF SUPPORT VECTOR MACHINES FOR FODDER CROP ASSESSMENT

Abstract. Identification of crop and its accuracy is an important aspect in predicting crop production using Remote Sensing technology. This study investigates the ability of Support Vector Machine (SVM) algorithm in discriminating fodder crops and estimating its area using moderate resolution multi-temporal Landsat-8 OLI data. SVM is a non-parametric statistical learning method and its accuracy is dependent on the parameters and the kernels used. The objective was to evaluate the feasibility of SVM in fodder classification and compare the results with traditional parametric Maximum Likelihood Classification (MLC). Fodder crops are available over small fields in the study area thus having large number of pure fodder pixels over small area is difficult. Hence, SVM has an advantage over MLC as it works well with less training data sets also. Three kernels (linear, polynomial and radial based function) were used with SVM classification. Comparative analysis showed that higher overall accuracy was observed in SVM in comparison to MLC. Temporal change in the spectral properties of the crops derived through Normalized Difference Vegetation Index (NDVI) from multi-temporal Landsat-8 was found to be the most important information that affects accuracy of classification. The classification accuracies for SVM with radial based function, polynomial, linear kernel and MLC were 90.09%, 89.9%, 88.9% and 82.4% respectively. The result suggested that SVM including three kernels performed significantly better than MLC. India has low livestock productivity due to unavailability of fodder hence this study could help in strengthening the fodder productivity.

Universal consistency and robustness of localized support vector machines

Abstract The massive amount of available data potentially used to discover patterns in machine learning is a challenge for kernel based algorithms with respect to runtime and storage capacities. Local approaches might help to relieve these issues. From a statistical point of view local approaches allow additionally to deal with different structures in the data in different ways. This paper analyses properties of localized kernel based, non-parametric statistical machine learning methods, in particular of support vector machines (SVMs) and methods close to them. We will show there that locally learnt kernel methods are universally consistent. Furthermore, we give an upper bound for the maxbias in order to show statistical robustness of the proposed method.

LPiTrack: Eye movement pattern recognition algorithm and application to biometric identification

A comprehensive nonparametric statistical learning framework, called LPiTrack, is introduced for large-scale eye-movement pattern discovery. The foundation of our data-compression scheme is based on a new Karhunen–Loéve-type representation of the stochastic process in Hilbert space by specially designed orthonormal polynomial expansions. We apply this novel nonlinear transformation-based statistical data-processing algorithm to extract temporal-spatial-static characteristics from eye-movement trajectory data in an automated, robust way for biometric authentication. This is a significant step towards designing a next-generation gaze-based biometric identification system. We elucidate the essential components of our algorithm through data from the second Eye Movements Verification and Identification Competition, organized as a part of the 2014 International Joint Conference on Biometrics.