Nonparametric Regression Techniques Research Articles

The Estimating Parameter and Number of Knots for Nonparametric Regression Methods in Modelling Time Series Data

This research aims to explore and compare several nonparametric regression techniques, including smoothing splines, natural cubic splines, B-splines, and penalized spline methods. The focus is on estimating parameters and determining the optimal number of knots to forecast cyclic and nonlinear patterns, applying these methods to simulated and real-world datasets, such as Thailand’s coal import data. Cross-validation techniques are used to control and specify the number of knots, ensuring the curve fits the data points accurately. The study applies nonparametric regression to forecast time series data with cyclic patterns and nonlinear forms in the dependent variable, treating the independent variable as sequential data. Simulated data featuring cyclical patterns resembling economic cycles and nonlinear data with complex equations to capture variable interactions are used for experimentation. These simulations include variations in standard deviations and sample sizes. The evaluation criterion for the simulated data is the minimum average mean square error (MSE), which indicates the most efficient parameter estimation. For the real data, monthly coal import data from Thailand is used to estimate the parameters of the nonparametric regression model, with the MSE as the evaluation metric. The performance of these techniques is also assessed in forecasting future values, where the mean absolute percentage error (MAPE) is calculated. Among the methods, the natural cubic spline consistently yields the lowest average mean square error across all standard deviations and sample sizes in the simulated data. While the natural cubic spline excels in parameter estimation, B-splines show strong performance in forecasting future values.

البرمجة الضبابية وتحسين أداء الخوارزميات في بيئة متعددة المتغيرات الرتيبة

In many situations of daily life, when we start statistical analysis, we encounter data that we suffer from inaccuracy in its measurement, either due to observation or measurement errors or the lack of appropriate conditions for collecting this data, such as failure times of machines, equipment, electrical devices, medical, engineering and economic data. In many medical studies, the researcher may have a strong conviction that the regression function used to describe the relationship between two variables has a certain form or formula that can be characterized by certain ordering restrictions such as monotonicity; monotonic regression has important applications in statistics, operations research, and signal processing, and these applications are often characterized by a very large value of n. For such large-scale problems, it is of great practical importance to develop algorithms whose complexity does not increase rapidly with increasing n. It is a nonparametric regression technique that preserves order. In this thesis, the proposed fuzzy autoregressive monotonic method, fuzzy autoregressive monotonic least squares (FILSR), fuzzy autoregressive M estimator (FIM), and fuzzy autoregressive maximum likelihood estimator (FIML) were proposed. The best method for estimating the autoregressive function is the fuzzy autoregressive maximum likelihood method with a mean square error of (0.51455).

S-wave log construction through semi-supervised regression clustering using machine learning: A case study of North Sea fields

Accurate prediction of S-wave velocity from well logs is essential for understanding subsurface geological formations and hydrocarbon reservoirs. Machine learning techniques, including clustering and regression, have emerged as effective methods for indirectly estimating S-wave logs and other rock properties. In this study, we employed clustering algorithms to identify similarities among well log datasets, encompassing depth, sonic, porosity, neutron, and apparent density, facilitating the discovery of correlations among various wells. These identified correlations served as a foundation for predicting S-wave values using a novel semi-supervised approach. Our approach combined clustering, specifically k-means clustering, with different types of regressors, including Least Squares Regression (LSR), Support Vector Regression (SVR), and Multi-Layer Perceptron (MLP). Our results demonstrate the superior performance of this integrated approach compared to traditional regression methods. We validated our methodology using various parametric and non-parametric regression techniques, showcasing its effectiveness not only on wells within the training region but also on wells outside the study area. We achieved a significant improvement in the R2 score metric, ranging from 2.22% to 6.51%, and a reduction in Mean Square Error (MSE) of at least 31% when compared to predictions made without clustering. This study underscores the potential of machine learning techniques for accurate prediction of S-wave velocity and other rock properties, thereby enhancing our comprehension of subsurface geology and hydrocarbon reservoirs.

Evaluation of Data Fitting Approaches for Speed/Flow Density Relationships

This paper presents guidance on data-fitting approaches in the context of pedestrian and evacuation dynamics research. In particular, it examines parametric and non-parametric regression techniques for analysing speed/flow density relationships. Parametric models assume predefined functional forms, while non-parametric models provide flexibility to capture complex relationships. This paper evaluates a range of traditional statistical approaches and machine-learning techniques. It emphasises the importance of weighting unbalanced datasets to enhance model accuracy. Practical applications are illustrated using traffic and pedestrian evacuation data. This paper is intended to stimulate discussion on best practices for developing, calibrating, and testing macroscopic and microscopic evacuation models. It does not prescribe a one-size-fits-all solution for evacuation data fitting approaches, but it provides an overview of existing methods and analyses their advantages and limitations.

Creating a Lactation Model for 305-Day Milk Yield with Different Resampling Techniques (Bagging Mars) in Mars Modeling

The main purpose of this research is to obtain a prediction model for milk yield by using Multivariate Adaptive Regression Splines (MARS) and Bagging MARS algorithms as a non-parametric regression technique. For this purpose, the effects on milk yield of 305 days were investigated by using lactation parameters in dairy cattle. In the study, 9337 lactation milk yield records belonging to 37 animals belonging to the 2022-2023 period were used and the data set was created by randomly ordering the animals. Data on milk yield results were analyzed with MARS and Bagging MARS algorithms. For dairy cattle; it was modeled with explanatory variables such as lactation month (month), service period (SP), last 7 days average milk yield (L7DMMY), animal's first birth age (FP), animal's age (Age), number of lactations (LN).Correlation coefficient (r), coefficient of determination (R2), Adjusted R2, Root of Square Mean Error (RMSE), standard deviation ratio (SD ratio), mean absolute percent error (MAPE), mean absolute for MARS algorithm estimating total average milk yield deviation (MAD) and Akaike Information Criteria (AIC) values are 0.9986, 0.997, 0.977, 0.142, 0.052, 0.2389, 0.086 and -88, respectively. Similar statistics for the Bagging MARS algorithm are 0.754, 0.556, 0.453, 1.8, 0.666, 3.96, 1.47, and 115, respectively. It has been observed that MARS and Bagging MARS algorithms provide correct results according to the goodness of fit statistics. In this study, it was revealed that MARS algorithm gave better results in milk yield modeling of 305-day lactation.

Co-registration and residual correction of digital elevation models: a comparative study

Abstract. Digital elevation models (DEMs) are currently one of the most widely used data sources in glacier thickness change research, due to the high spatial resolution and continuous coverage. However, raw DEM data are often misaligned with each other, due to georeferencing errors, and a co-registration procedure is required before DEM differencing. In this paper, we present a comparative analysis of the two classical co-registration methods proposed by Nuth and Kääb (2011) and Rosenholm and Torlegard (1988). The former is currently the most commonly used method in glacial studies, while the latter is a seminal work in the photogrammetric field that has not been extensively investigated by the cryosphere community. Furthermore, we also present a new residual correction method using a generalized additive model (GAM) to eliminate the remaining systematic errors in DEM co-registration results. The performance of the two DEM co-registration methods and three residual correction algorithms (the GAM-based method together with two parametric-model-based methods) was evaluated using multiple DEM pairs from the Greenland Ice Sheet and mountain glaciers, including Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) DEMs, ZiYuan-3 (ZY-3) DEMs, the Shuttle Radar Topography Mission (SRTM) DEM, and the Copernicus DEM. The experimental results confirm our theoretical analysis of the two co-registration methods. The method of Rosenholm and Torlegard has a greater ability to remove DEM misalignments (an average of 4.6 % and 13.7 % for the test datasets from Greenland Ice Sheet and High Mountain Asia, respectively) because it models the translation, scale, and rotation-induced biases, while the method of Nuth and Kääb considers translation only. The proposed GAM-based method performs statistically better than the two residual correction methods based on parametric regression models (high-order polynomials and the sum of the sinusoidal functions). A visual inspection reveals that the GAM-based method, as a non-parametric regression technique, can capture complex systematic errors in the DEM co-registration residuals.

Macro Stress Testing the Credit Risk of Conventional and Participation Banks in Turkey: A Nonparametric Quantile Regression Approach

ABSTRACT This paper applies a novel time series-based additive nonparametric quantile regression technique to stress test the credit risk of conventional and participation banks in Turkey. We particularly examine the effects of the exchange rate, unemployment rate, the policy interest rate, and the public debt-to-GDP ratio on the non-performing loan ratio (NPL) of Turkish banks at distinct quantile levels. We find that their effects are heterogeneous and significantly vary across different types of banks and quantiles. Overall, participation banks are more vulnerable to negative scenarios associated with the exchange rate, unemployment rate, and public debt ratio than conventional banks.

Bayesian neural tree models for nonparametric regression

SummaryFrequentist and Bayesian methods differ in many aspects but share some basic optimal properties. In real‐life prediction problems, situations exist in which a model based on one of the above paradigms is preferable depending on some subjective criteria. Nonparametric classification and regression techniques, such as decision trees and neural networks, have both frequentist (classification and regression trees (CARTs) and artificial neural networks) as well as Bayesian counterparts (Bayesian CART and Bayesian neural networks) to learning from data. In this paper, we present two hybrid models combining the Bayesian and frequentist versions of CART and neural networks, which we call the Bayesian neural tree (BNT) models. BNT models can simultaneously perform feature selection and prediction, are highly flexible, and generalise well in settings with limited training observations. We study the statistical consistency of the proposed approaches and derive the optimal value of a vital model parameter. The excellent performance of the newly proposed BNT models is shown using simulation studies. We also provide some illustrative examples using a wide variety of standard regression datasets from a public available machine learning repository to show the superiority of the proposed models in comparison to popularly used Bayesian CART and Bayesian neural network models.

Quantile Regression in Space-Time Varying Coefficient Model of Upper Respiratory Tract Infections Data

Space-time varying coefficient models, which are used to identify the effects of covariates that change over time and spatial location, have been widely studied in recent years. One such model, called the quantile regression model, is particularly useful when dealing with outliers or non-standard conditional distributions in the data. However, when the functions of the covariates are not easily specified in a parametric manner, a nonparametric regression technique is often employed. One such technique is the use of B-splines, a nonparametric approach used to estimate the parameters of the unspecified functions in the model. B-splines smoothing has potential to overfit when the number of knots is increased, and thus, a penalty is added to the quantile objective function known as P-splines. The estimation procedure involves minimizing the quantile loss function using an LP-Problem technique. This method was applied to upper respiratory tract infection data in the city of Bandung, Indonesia, which were measured monthly across 30 districts. The results of the study indicate that there are differences in the effect of covariates between quantile levels for both space and time coefficients. The quantile curve estimates also demonstrate robustness with respect to outliers. However, the simultaneous estimation of the quantile curves produced estimates that were relatively close to one another, meaning that some quantile curves did not depict the actual data pattern as precisely. This suggests that each district in Bandung City not only has different categories of incidence rates but also has a heterogeneous incidence rate based on three quantile levels, due to the difference in the effects of covariates over time and space.

Appraisal of machine learning techniques for predicting emerging disinfection byproducts in small water distribution networks

Monitoring emerging disinfection byproducts (DBPs) is challenging for many small water distribution networks (SWDNs), and machine learning-based predictive modeling could be an alternative solution. In this study, eleven machine learning techniques, including three multivariate linear regression-based, three regression tree-based, three neural networks-based, and two advanced non-parametric regression techniques, are used to develop models for predicting three emerging DBPs (dichloroacetonitrile, chloropicrin, and trichloropropanone) in SWDNs. Predictors of the models include commonly-measured water quality parameters and two conventional DBP groups. Sampling data of 141 cases were collected from eleven SWDNs in Canada, in which 70 % were randomly selected for model training and the rest were used for validation. The modeling process was reiterated 1000 times for each model. The results show that models developed using advanced regression techniques, including support vector regression and Gaussian process regression, exhibited the best prediction performance. Support vector regression models showed the highest prediction accuracy (R2 =0.94) and stability for predicting dichloroacetonitrile and trichloropropanone, and Gaussian process regression models are optimal for predicting chloropicrin (R2 =0.92). The difference is likely due to the much lower concentrations of chloropicrin than dichloroacetonitrile and trichloropropanone. Advanced non-parametric regression techniques, characterized by a probabilistic nature, were identified as most suitable for developing the predictive models, followed by neural network-based (e.g., generalized regression neural network), regression tree-based (e.g., random forest), and multivariate linear regression-based techniques. This study identifies promising machine learning techniques among many commonly-used alternatives for monitoring emerging DBPs in SWDNs under data constraints.

Principled Machine Learning

We introduce the underlying concepts which give rise to some of the commonly used machine learning methods, excluding deep-learning machines and neural networks. We point to their advantages, limitations and potential use in various areas of photonics. The main methods covered include parametric and non-parametric regression and classification techniques, kernel-based methods and support vector machines, decision trees, probabilistic models, Bayesian graphs, mixture models, Gaussian processes, message passing methods and visual informatics.

Nonparametric multiplicative distortion measurement errors models with bias reduction

In this paper we propose a new nonparametric regression technique under multiplicative distortion measurement errors settings. The unobservable variables are both distorted in a multiplicative fashion by an observed confounding variable. A bias reduction is proposed by choosing a function that is the projection of the unknown regression function onto the parametric family in a certain metric. We find that this new technique leads to substantial improvement in the performance of regression estimators in comparison with the direct one-step estimation, irrespective of the choice of a parametric model. We obtain asymptotic normality results for the estimated nonparametric kernel smoothers, and further discuss their estimation efficiency. We conduct Monte Carlo simulation experiments to examine the performance of the proposed estimators.

Assessment of the change of trend in precipitation over Afghanistan in 1979–2019

The civil war, harsh climate, tough topography, and lack of accurate meteorological stations have limited the number of consecutive synoptic data across Afghanistan. The global data (gridded precipitation datasets) pave the way to assess the precipitation indicators of climate, where stations are sparsely located. This study assessed the mean annual precipitation trend in 33 stations over Afghanistan. Non-parametric linear regression technique was employed to find upward and downward trends and magnitudes. The daily of precipitation was obtained from the database of the CPC-NOAA (Climate Prediction Center - National Oceanic Atmospheric Administration) for the period of 1979–2019. The CPC spatial resolution of daily precipitation is 0.5×0.5 degree. Analysis of mean annual precipitation showed a significant decreasing trend at six provinces in the north, while an increasing trend of 9.2 mm per decade has been observed at three provinces. In the south, a notable reduction of the precipitation trend has been experienced in Helmand, Kandahar, and Nimruz provinces, but Ghazni and Uruzgan show a positive trend. Data revealed that mean annual precipitation has remarkably decreased in the western part of Afghanistan. According to the study period, the mean annual rainfall in the central regions indicates a raise of 37.5 mm per decade in Kabul, while in Vardak, the precipitation increases up to 9.21 mm per year. Eastern regions include 8 provinces, and the eastern highland covers the smallest area that is mainly covered by rangeland and the largest existing forests. These regions are directly influenced by the moist air masses of Indian monsoon getting trapped at the high mountain slopes, and it can lead to an increase of rain. Data reveals an upward trend of precipitation in the eastern part of Afghanistan.

On Quantiles Estimation Based on Stratified Sampling Using Multiplicative Bias Correction Approach

In the context of stratified sampling, we develop a nonparametric regression technique to estimating finite population quantiles in model‐based frameworks using a multiplicative bias correction strategy. Furthermore, the proposed estimator’s asymptotic behavior is presented, and when certain conditions are met, the estimator is observed to be asymptotically unbiased and asymptotically consistent. Simulation studies were conducted to determine the proposed estimator’s performance for the three quartiles of two fictitious populations under varied distributional assumptions. Based on relative biases, mean‐squared errors, and relative root‐mean‐squared errors, the proposed estimator can be extremely satisfactory, according to these findings.

Prediction of transportation energy demand: Multivariate Adaptive Regression Splines

Energy usage in the transportation sector has been increasing in Turkey. Good management of energy is important as well as a reliable prediction of the energy demand in the transportation sector. The main objective of this research is to predict transport energy demand using Multivariate Adaptive Regression Splines (MARS) as a nonparametric regression technique. Transport energy demand was modeled for the period 1975–2019 based on a mix of factors including the gross domestic product (GDP), population, vehicle-km, ton-km, passenger-km and oil price. Five models were established and compared with real data collected from the Ministry of Energy and Natural Resources (MENR). Five MARS models including pairs of predictors, i.e. oil price-GDP, oil price-population, oil price-ton, oil price-vehicle and oil price-passenger, were evaluated comparatively in the prediction of energy demand. Among the candidate models, the third MARS model, which had the lowest RMSE, SD ratio, AICc values and the highest R2, Adjusted R2 and especially GR2 value, was selected as the best predictive model. In conclusion, it could be suggested that the third MARS model produced the highest predictive performance in the prediction of energy demand by two predictors, ton and oil price.

Parameter Estimation of Multivariate Adaptive Regression Spline (MARS) with Stepwise Approach to Multi Drug-Resistant Tuberculosis (MDR-TB) Modeling in Lamongan Regency

Tuberculosis (TB) is an infectious disease, caused by mycobacterium tuberculosis that affects various organs, especially the lungs. If TB treatment is not done thoroughly by the patient, then it can lead to death. Tuberculosis disease could not be healed cause TB bacteria to double immunity against anti-TB drugs, called multi-drug resistant (MDR). One identification issue toward TB infection chain is a TB case distribution analysis using mathematical modeling. The method used in this study was Multivariate Adaptive Regression Spline (MARS). Multivariate Adaptive Regression Spline is one type of non-parametric regression techniques, where the model does not assume the functional relationship between response and predictor variables, and has a flexible functional structure as well. Modeling aims to determine the factors that have the most significant influence on MDR-TB cases in Lamongan regency, as well as predict the incidence of MDR-TB in each sub-regency. The results show, the best model has a combination of BF = 28, MI = 2 and MO = 3, based on the minimum GCV, which is 5.26E-06. Furthermore, the model is statistically proper according to the criteria of APER and Press’s Q.

Simultaneous hedging strategy for price and volume risks in electricity businesses using energy and weather derivatives

In general, electric utilities face intrinsic risks as their revenues depend on high volatility factors including price and volume of sales/procurements. Aiming for an effective strategy to control those risks, we construct a hedging portfolio based on energy and weather derivatives, which can minimize the revenue fluctuations. To this end, we provide unique methods by applying nonparametric regression techniques to synthesize the payoff functions of derivatives that change with time, based on tensor product spline functions. The proposed methodology enables us to incorporate two dimensional smoothing conditions of the underlying asset price and expiration date with a yearly cyclical trend. Moreover, we show that the applied method of Analysis of Variance (ANOVA) decomposition can separate deterministic time trends from the original multivariate payoff functions, and hence, a simultaneous estimation of multiple derivatives payoff functions is achieved. By assuming that revenues have yearly cyclical trends even when viewed at the rate of annual change, we also introduce a spline function with cross variables to consider such a mixed effect. In addition, we propose new standardized derivatives with the square of the temperature prediction error as the underlying asset. Empirical analysis using data from both the Japan Electric Power Exchange (JEPX) and PJM in the U.S. demonstrates the significant hedging effect and supports the versatility of the proposed modeling approach.

LOESS radiometric correction for contiguous scenes (LORACCS): Improving the consistency of radiometry in high-resolution satellite image mosaics

Analyses of Planet Dove (CubeSat) satellite imagery can prove challenging when radiometric consistency among contiguous scenes is necessary to apply algorithms across mosaics or through time. While temporal resolution is excellent due to the high number of Dove satellites, the radiometric consistency between scenes can be a limiting factor in the utility of the imagery when creating mosaics. To date, methods of addressing the radiometric consistency problem have focused on radiometric correction in a time-series stack, but do not address how to achieve seamless mosaics within that stack. We demonstrate a method to correct Planet Dove radiometric inconsistencies in mosaics using locally estimated scatterplot smoothing (LOESS). LOESS is a non-parametric regression technique that is more flexible to fitting a line to data than ordinary least squares regression (OLS), and better conforms to the nonlinearities in spectral responses among Dove sensors across brightness levels. A straightforward application of LOESS uses overlap areas between contiguous scenes, enabling correction independent of outside data sources. We demonstrate the effectiveness of the method to improve the quality of mosaics by reducing spectral discontinuities between overlapping images in multiple locations across multiple biome and landcover types. Further, the LOESS Radiometric Correction for Contiguous Scenes, or LORACCS, is able to be integrated with methods previously developed to normalize Dove imagery with sensors such as Landsat or Sentinel-2, typically for use in time series analysis. Written in Python, this method offers an open-source solution to a significant challenge when working with Dove data.

Quantile regression in varying coefficient model of upper respiratory tract infections in Bandung City

Varying coefficient models are commonly used to obtain effects of covariates that vary over other variables. A special case of varying coefficient model is applied to longitudinal data where the covariates may vary over time. When the function is not easy to specify parametrically, we then need to work on a non-parametric regression technique. In this case, we approximate the function by B-splines. B-splines smoothing tends to overfit with increasing knots, then a penalty is added to the quantile objective function. This estimation procedure is called P-splines. As the objective function, we propose to use quantile loss function. The technique will be implemented to the upper respiratory tract infection data in Bandung City which was measured repeatedly from 30 sub district in Bandung City and hence we have a longitudinal data structure.

Using phase-specific nonparametric density regression technique to model and compare the COVID-19 infection rate between South Korea and Italy

Using phase-specific nonparametric density regression technique to model and compare the COVID-19 infection rate between South Korea and Italy