Random K-fold Cross-validation Research Articles

Random forests with spatial proxies for environmental modelling: opportunities and pitfalls

Abstract. Spatial proxies, such as coordinates and distance fields, are often added as predictors in random forest (RF) models without any modifications being made to the algorithm to account for residual autocorrelation and improve predictions. However, their suitability under different predictive conditions encountered in environmental applications has not yet been assessed. We investigate (1) the suitability of spatial proxies depending on the modelling objective (interpolation vs. extrapolation), the strength of the residual spatial autocorrelation, and the sampling pattern; (2) which validation methods can be used as a model selection tool to empirically assess the suitability of spatial proxies; and (3) the effect of using spatial proxies in real-world environmental applications. We designed a simulation study to assess the suitability of RF regression models using three different types of spatial proxies: coordinates, Euclidean distance fields (EDFs), and random forest spatial prediction (RFsp). We also tested the ability of probability sampling test points, random k-fold cross-validation (CV), and k-fold nearest neighbour distance matching (kNNDM) CV to reflect the true prediction performance and correctly rank models. As real-world case studies, we modelled annual average air temperature and fine particulate air pollution for continental Spain. In the simulation study, we found that RFs with spatial proxies were poorly suited for spatial extrapolation to new areas due to significant feature extrapolation. For spatial interpolation, proxies were beneficial when both strong residual autocorrelation and regularly or randomly distributed training samples were present. In all other cases, proxies were neutral or counterproductive. Random k-fold cross-validation generally favoured models with spatial proxies even when it was not appropriate, whereas probability test samples and kNNDM CV correctly ranked models. In the case studies, air temperature stations were well spread within the prediction area, and measurements exhibited strong spatial autocorrelation, leading to an effective use of spatial proxies. Air pollution stations were clustered and autocorrelation was weaker and thus spatial proxies were not beneficial. As the benefits of spatial proxies are not universal, we recommend using spatial exploratory and validation analyses to determine their suitability, as well as considering alternative inherently spatial modelling approaches.

Evaluation of Building Seismic Capacity Based on Improved Naive Bayesian Algorithm

The influencing factors of building seismic capacity are analyzed, the basic cause events of the assessment target based on fault tree analysis (FTA) are determined, the basic cause events in the FTA model are classified and summarized, and a judgment system of building seismic capacity is built. The weight of each index factor in the Gini index calculation system is used, and the importance of the index is analyzed. On the basis of the Spearman correlation coefficient calculation of the index, the improved naive Bayesian algorithm is combined with the importance of the index to build a judgment model for the seismic capacity of housing buildings. The sample set is constructed based on the judgment system with the basic data of some housing buildings in Huoshan County. In order to improve the generalization ability and avoid overfitting, the K-SMOTE algorithm for mixed sampling was modified to improve sample balance, and random k -fold cross validation method was used for sample division and model optimization, achieving the determination of seismic capacity level of building. The research results indicate the following: (1) the accuracy of model evaluation is 93%, with model accuracy and recall rates of 0.913 and 0.93, respectively, indicating strong generalization ability of the model. (2) Selecting some actual examples of a building, the model judgment results are consistent with the actual results, verifying the correctness of the proposed method for building the model, which can be effectively used for determining the seismic capacity of building structures. (3) Applying the proposed method to the seismic capacity assessment of buildings in the Ta-pieh Mountains of Lu’an, it is concluded that the seismic capacity of urban buildings is common, while that of rural buildings is poor.

Analysis of longitudinal data using constrained repeated random sampling-cross validation (CORRS-CV) and partial least squares

Longitudinal data constitutes a very important source of information to study systems or individuals over time. Analysis of this type of data is often performed using multivariate models to assess the association between high-dimensionality data and an independent variable for e.g., the identification of biomarkers of disease, building accurate classifiers, or the identification of descriptors of a time-dependent process. Cross validation (CV) is frequently used for model selection and development, but current CV strategies such as random k-fold CV or individual-based CV can provide overly optimistic model performance estimates because the independence between the samples included in the train and test subsets is often not ensured. To overcome this potential pitfall, here we show with help of simulated data how the use of constrained repeated random subsampling – cross validation (CORRS-CV) improves the independence between train and test subsets during CV, thus providing an accurate estimation of the model performance and facilitating the identification of informative variables.

COVID-19 mortality analysis from soft-data multivariate curve regression and machine learning.

A multiple objective space-time forecasting approach is presented involving cyclical curve log-regression, and multivariate time series spatial residual correlation analysis. Specifically, the mean quadratic loss function is minimized in the framework of trigonometric regression. While, in our subsequent spatial residual correlation analysis, maximization of the likelihood allows us to compute the posterior mode in a Bayesian multivariate time series soft-data framework. The presented approach is applied to the analysis of COVID-19 mortality in the first wave affecting the Spanish Communities, since March 8, 2020 until May 13, 2020. An empirical comparative study with Machine Learning (ML) regression, based on random k-fold cross-validation, and bootstrapping confidence interval and probability density estimation, is carried out. This empirical analysis also investigates the performance of ML regression models in a hard- and soft-data frameworks. The results could be extrapolated to other counts, countries, and posterior COVID-19 waves.Supplementary InformationThe online version contains supplementary material available at 10.1007/s00477-021-02021-0.

Learning From Limited Data: Towards Best Practice Techniques for Antimicrobial Resistance Prediction From Whole Genome Sequencing Data.

Antimicrobial resistance prediction from whole genome sequencing data (WGS) is an emerging application of machine learning, promising to improve antimicrobial resistance surveillance and outbreak monitoring. Despite significant reductions in sequencing cost, the availability and sampling diversity of WGS data with matched antimicrobial susceptibility testing (AST) profiles required for training of WGS-AST prediction models remains limited. Best practice machine learning techniques are required to ensure trained models generalize to independent data for optimal predictive performance. Limited data restricts the choice of machine learning training and evaluation methods and can result in overestimation of model performance. We demonstrate that the widely used random k-fold cross-validation method is ill-suited for application to small bacterial genomics datasets and offer an alternative cross-validation method based on genomic distance. We benchmarked three machine learning architectures previously applied to the WGS-AST problem on a set of 8,704 genome assemblies from five clinically relevant pathogens across 77 species-compound combinations collated from public databases. We show that individual models can be effectively ensembled to improve model performance. By combining models via stacked generalization with cross-validation, a model ensembling technique suitable for small datasets, we improved average sensitivity and specificity of individual models by 1.77% and 3.20%, respectively. Furthermore, stacked models exhibited improved robustness and were thus less prone to outlier performance drops than individual component models. In this study, we highlight best practice techniques for antimicrobial resistance prediction from WGS data and introduce the combination of genome distance aware cross-validation and stacked generalization for robust and accurate WGS-AST.

Himawari-8 Aerosol Optical Depth (AOD) Retrieval Using a Deep Neural Network Trained Using AERONET Observations

Spectral aerosol optical depth (AOD) estimation from satellite-measured top of atmosphere (TOA) reflectances is challenging because of the complicated TOA-AOD relationship and a nexus of land surface and atmospheric state variations. This task is usually undertaken using a physical model to provide a first estimate of the TOA reflectances which are then optimized by comparison with the satellite data. Recently developed deep neural network (DNN) models provide a powerful tool to represent the complicated relationship statistically. This study presents a methodology based on DNN to estimate AOD using Himawari-8 Advanced Himawari Imager (AHI) TOA observations. A year (2017) of AHI TOA observations over the Himawari-8 full disk collocated in space and time with Aerosol Robotic Network (AERONET) AOD data were used to derive a total of 14,154 training and validation samples. The TOA reflectance in all six AHI solar bands, three TOA reflectance ratios derived based on the dark-target assumptions, sun-sensor geometry, and auxiliary data are used as predictors to estimate AOD at 500 nm. The DNN AOD is validated by separating training and validation samples using random k-fold cross-validation and using AERONET site-specific leave-one-station-out validation, and is compared with a random forest regression estimator and Japan Meteorological Agency (JMA) AOD. The DNN AOD shows high accuracy: (1) RMSE = 0.094, R2 = 0.915 for k-fold cross-validation, and (2) RMSE = 0.172, R2 = 0.730 for leave-one-station-out validation. The k-fold cross-validation overestimates the DNN accuracy as the training and validation samples may come from the same AHI pixel location. The leave-one-station-out validation reflects the accuracy for large-area applications where there are no training samples for the pixel location to be estimated. The DNN AOD has better accuracy than the random forest AOD and JMA AOD. In addition, the contribution of the dark-target derived TOA ratio predictors is examined and confirmed, and the sensitivity to the DNN structure is discussed.

Assessment of machine learning classifiers for global lake ice cover mapping from MODIS TOA reflectance data

The topic of satellite remote sensing of lake ice has gained considerable attention in recent years. Optical satellite data from the Moderate Resolution Imaging Spectroradiometer (MODIS) allow for the monitoring of lake ice cover (an Essential Climate Variable or ECV), and dates associated with ice phenology (freeze-up, break-up, and ice cover duration) over large areas in an era where ground-based observational networks have nearly vanished in many northern countries. Ice phenology dates as well as dates of maximum and minimum ice cover extent (for lakes that do not form a complete ice cover in winter or do not totally lose their ice cover in summer) are useful for assessing long-term trends and variability in climate, particularly due to their sensitivity to changes in near-surface air temperature. Existing knowledge-driven (threshold-based) retrieval algorithms for lake ice cover mapping that use top-of-atmosphere (TOA) reflectance products do not perform well under lower solar illumination conditions (i.e. large solar zenith angles), resulting in low TOA reflectance. This research assessed the capability of four machine learning classifiers (i.e. multinomial logistic regression, MLR; support vector machine, SVM; random forest, RF; gradient boosting trees, GBT) for mapping lake ice cover, water and cloud cover during both break-up and freeze-up periods using the MODIS/Terra L1B TOA (MOD02) product. The classifiers were trained and validated using samples collected from 17 large lakes across the Northern Hemisphere (Europe and North America); lakes that represent different characteristics with regards to area, latitude, freezing frequency, and ice duration. Following an accuracy assessment using random k-fold cross-validation (k = 100), all machine learning classifiers using a 7-band combination (visible, near-infrared and shortwave-infrared) were found to be able to produce overall classification accuracies above 94%. Both RF and GBT provided overall and class-specific accuracies above 98% and a more visually accurate depiction of lake ice, water and cloud cover. The two tree-based classifiers offered the most robust spatial transferability over the 17 lakes and performed consistently well across ice seasons. However, only RF was relatively insensitive to the choice of the hyperparameters compared to the other three classifiers. The results demonstrate the potential of RF for mapping lake ice cover globally from MODIS TOA reflectance data.

Modeling and mapping aboveground biomass of the restored mangroves using ALOS-2 PALSAR-2 in East Kalimantan, Indonesia

Accurate estimation of forest aboveground biomass (AGB) using remote sensing is a requisite for monitoring, reporting and verification (MRV) system of the United Nations Programme on Reducing Emissions from Deforestation and Forest Degradation. However, attaining high accuracy remains a great challenge in the diverse tropical forests. Among available technologies, l-band Synthetic Aperture Radar (SAR) estimates AGB with reasonably high accuracy in the terrestrial tropical forests. Nevertheless, the accuracy is relatively low in the mangrove forests. In this context, the study was carried out to model and map AGB using backscatter coefficients of Advanced Land Observing Satellite-2 (ALOS-2) Phased Array l-band SAR-2 (PALSAR-2) in part of the restored mangrove forest at Mahakam Delta, Indonesia. PALSAR-2 data was acquired with image scene observation during the peak low tide on 30 July 2018 from Japan Aerospace Exploration Agency. The forest parameters namely tree height and diameter at breast height were measured from 71 field plots in September-October 2018. The parameters were used in mangrove allometry to calculate the field AGB. Finally, HV polarized backscatter coefficients of PALSAR-2 were used to model AGB using linear regression. The model demonstrated a comparatively high performance using three distinct methods viz. independent validation (R2 of 0.89 and RMSE of 23.16 tons ha−1), random k-fold cross validation (R2 of 0.89 and RMSE of 24.59 tons ha−1) and leave location out cross validation (LLO CV) (R2 of 0.88 and RMSE of 24.05 tons ha−1). The high accuracy of the LLO CV indicates no spatial overfitting in the model. Thus, the model based on LLO CV was used to map AGB in the study area. This is the first study that successfully obtains high accuracy in modeling AGB in the mangrove forest. Therefore, it offers a significant contribution to the MRV mechanism for monitoring mangrove forests in the tropics and sub-tropics.

Entropy and mutual information in genome-wide selection: the splitting of k-fold cross-validation sets and implications for tree breeding

Random k-fold cross-validation in genome-wide selection (GWS) can help to estimate predictive ability ($$ {r}_{y\hat{y}} $$). Predictive ability tends to be higher when training, and validation sets present a high degree of kinship. However, many tree breeding populations are less genetically related to the training sets and have different levels of phenotypic diversity. Therefore, this study proposes methods of splitting k-fold cross-validation sets to optimize $$ {r}_{y\hat{y}} $$ estimates that are consistent with the breeding population and verify the impact of phenotypic and genotypic distribution on GWS. Using a simulated Eucalyptus trait (h2=0.5) and Pinus taeda L. data for diameter at breast height (h2=0.31), six methods were developed based on mutual information (I) and entropy (H) for measuring genetic similarity and phenotypic dissimilarity, respectively. All methods were evaluated for $$ {r}_{y\hat{y}} $$, bias, minimum squared error of prediction, and genomic heritability. The Pearson correlations of these parameters with the kinship coefficient, and I and H between and within training and validation sets were also estimated. Our results show that closer genetic similarity did not significantly increase $$ {r}_{y\hat{y}} $$ and that a lower H reduced $$ {r}_{y\hat{y}} $$ and overestimated genomic breeding values. Consequently, phenotypic diversity (high H) should be added to tree breeding populations to increase genetic gain and reduce bias. The new methods accurately fitted models according to the entropy of tree breeding populations and their genetic relationship to the training sets. Therefore, these methods provided usable estimates of genetic gain to produce consistent success of long-term tree breeding programs.

Improving performance of spatio-temporal machine learning models using forward feature selection and target-oriented validation

Importance of target-oriented validation strategies for spatio-temporal prediction models is illustrated using two case studies: (1) modelling of air temperature (Tair) in Antarctica, and (2) modelling of volumetric water content (VW) for the R.J. Cook Agronomy Farm, USA. Performance of a random k-fold cross-validation (CV) was compared to three target-oriented strategies: Leave-Location-Out (LLO), Leave-Time-Out (LTO), and Leave-Location-and-Time-Out (LLTO) CV. Results indicate that considerable differences between random k-fold (R2 = 0.9 for Tair and 0.92 for VW) and target-oriented CV (LLO R2 = 0.24 for Tair and 0.49 for VW) exist, highlighting the need for target-oriented validation to avoid an overoptimistic view on models. Differences between random k-fold and target-oriented CV indicate spatial over-fitting caused by misleading variables. To decrease over-fitting, a forward feature selection in conjunction with target-oriented CV is proposed. It decreased over-fitting and simultaneously improved target-oriented performances (LLO CV R2 = 0.47 for Tair and 0.55 for VW).

Nonparametric criteria for supervised classification of fuzzy data

The supervised classification of fuzzy data obtained from a random experiment is discussed. The data generation process is modelled through random fuzzy sets which, from a formal point of view, can be identified with certain function-valued random elements. First, one of the most versatile discriminant approaches in the context of functional data analysis is adapted to the specific case of interest. In this way, discriminant analysis based on nonparametric kernel density estimation is discussed. In general, this criterion is shown not to be optimal and to require large sample sizes. To avoid such inconveniences, a simpler approach which eludes the density estimation by considering conditional probabilities on certain balls is introduced. The approaches are applied to two experiments; one concerning fuzzy perceptions and linguistic labels and another one concerning flood analysis. The methods are tested against linear discriminant analysis and random K-fold cross validation.