Training Set Design Research Articles

Deep learning insights into non-universality in the halo mass function

ABSTRACT The abundance of dark matter haloes is a key cosmological probe in forthcoming galaxy surveys. The theoretical understanding of the halo mass function (HMF) is limited by our incomplete knowledge of the origin of non-universality and its cosmological parameter dependence. We present a deep-learning model which compresses the linear matter power spectrum into three independent factors which are necessary and sufficient to describe the $z=0$ HMF from the state-of-the-art Aemulus emulator to sub-per cent accuracy in a wCDM$+N_\mathrm{eff}$ parameter space. Additional information about growth history does not improve the accuracy of HMF predictions if the matter power spectrum is already provided as input, because required aspects of the former can be inferred from the latter. The three factors carry information about the universal and non-universal aspects of the HMF, which we interrogate via the information-theoretic measure of mutual information. We find that non-universality is captured by recent growth history after matter-dark-energy equality and $N_{\rm eff}$ for $M\sim 10^{13} \, \mathrm{M_\odot }\, h^{-1}$ haloes, and by $\Omega _{\rm m}$ for $M\sim 10^{15} \, \mathrm{M_\odot }\, h^{-1}$. The compact representation learnt by our model can inform the design of emulator training sets to achieve high emulator accuracy with fewer simulations.

Goal‐Oriented Two‐Layered Kernel Models as Automated Surrogates for Surface Kinetics in Reactor Simulations

AbstractMulti‐scale modeling allows the description of real reactive systems under industrially relevant conditions. However, its application to rational catalyst and reactor design is hindered by the prohibitively high computational cost associated with the chemical kinetics on the catalyst scale. Here, the computational cost is drastically reduced by introducing goal‐oriented kernel models that serve as surrogates for the chemical kinetics. This special model type allows for automated training set design and reliable results, even outside the training region. Therefore, it can be envisioned as a plug‐and‐play solution for accelerating reactive flow simulations with guaranteed accuracy.

A kind of numerical model combined with genetic algorithm and back propagation neural network for creep-fatigue life prediction and optimization of double-layered annulus metal hydride reactor and verification of ASME-NH code

The safety and stability of metal hydride reactor (MHR) structure are significantly important for normal operations of hydrogen isotope storage and supply system. Moreover, considering that the experiments of studying the fatigue and creep properties of MHR structure are always time, money, difficulty and energy consuming. To further improve the service life of MHR as well as lower the failure risk, this paper proposes a system integration method based on ASME code validation, which combines genetic algorithm (GA) and back propagation neural network (BPNN) for rapid assessment of creep-fatigue life (CFL) and optimal design of structural parameters. First, numerical simulation is applied to perform thermal-mechanical coupled finite element analysis and verified by ASME code. Then, parametric sensitivity analysis and experimental design of parameter-based training set based on Taguchi method were performed, and response values are obtained from physical models combined with numerical simulations. Finally, a parametric optimization procedure combining BPNN and GA is developed based on the training data. In the workflow, parametric data flow is passed through the experimental design, numerical simulation, prediction and optimization processes. The results show that the proposed BPNN proxy model based on ASME code validation has good performance in predicting the life of MHR. Based on this, the life of the MHR is improved by 8% compared to the reference sample.

Genomic prediction in hybrid breeding: I. Optimizing the training set design

Key messageTraining sets produced by maximizing the number of parent lines, each involved in one cross, had the highest prediction accuracy for H0 hybrids, but lowest for H1 and H2 hybrids.Genomic prediction holds great promise for hybrid breeding but optimum composition of the training set (TS) as determined by the number of parents (nTS) and crosses per parent (c) has received little attention. Our objective was to examine prediction accuracy (r_{a}) of GCA for lines used as parents of the TS (I1 lines) or not (I0 lines), and H0, H1 and H2 hybrids, comprising crosses of type I0 × I0, I1 × I0 and I1 × I1, respectively, as function of nTS and c. In the theory, we developed estimates for r_{a} of GBLUPs for hybrids: (i)hat{r}_{a} based on the expected prediction accuracy, and (ii) tilde{r}_{a} based on r_{a} of GBLUPs of GCA and SCA effects. In the simulation part, hybrid populations were generated using molecular data from two experimental maize data sets. Additive and dominance effects of QTL borrowed from literature were used to simulate six scenarios of traits differing in the proportion (τSCA = 1%, 6%, 22%) of SCA variance in σG2 and heritability (h2 = 0.4, 0.8). Values of tilde{r}_{a} and hat{r}_{a} closely agreed with r_{a} for hybrids. For given size NTS = nTS × c of TS, r_{a} of H0 hybrids and GCA of I0 lines was highest for c = 1. Conversely, for GCA of I1 lines and H1 and H2 hybrids, c = 1 yielded lowest r_{a} with concordant results across all scenarios for both data sets. In view of these opposite trends, the optimum choice of c for maximizing selection response across all types of hybrids depends on the size and resources of the breeding program.

Training set designs for prediction of yield and moisture of maize test cross hybrids with unreplicated trials.

Unreplicated field trials and genomic prediction are both used to enhance the efficiency in early selection stages of a hybrid maize breeding program. No results are available on the optimal experimental design when combining both approaches. Our objectives were to investigate the effect of the training set design on the accuracy of genomic prediction in unreplicated maize test crosses. We carried out a cross validation study on basis of an experimental data set consisting of 1436 hybrids evaluated for yield and moisture for which genotyping information of 461 SNP markers were available. Training set designs of different size, implementing within environment prediction, within year prediction, across year prediction, and combinations of data sources across years and environments were compared with respect to their prediction accuracy. Across year prediction did not reach prediction accuracies that are useful for genomic selection. Within year prediction across environments provided useful correlations between observed and predicted breeding values. The prediction accuracies did not improve when adding to the training set data from previous years. We conclude that using all data available from unreplicated tests of the current breeding cycle provides a good accuracy of predicting test crosses, whereas adding data from previous breeding cycles, in which the genotypes are less related to the tested material, has only limited value for increasing the prediction accuracy.

Genomic prediction of tocochromanols in exotic-derived maize.

Tocochromanols (vitamin E) are an essential part of the human diet. Plant products, including maize (Zea mays L.) grain, are the major dietary source of tocochromanols; therefore, breeding maize with higher vitamin content (biofortification) could improve human nutrition. Incorporating exotic germplasm in maize breeding for trait improvement including biofortification is a promising approach and an important research topic. However, information about genomic prediction of exotic-derived lines using available training data from adapted germplasm is limited. In this study, genomic prediction was systematically investigated for nine tocochromanol traits within both an adapted (Ames Diversity Panel [AP]) and an exotic-derived (Backcrossed Germplasm Enhancement of Maize [BGEM]) maize population. Although prediction accuracies up to 0.79 were achieved using genomic best linear unbiased prediction (gBLUP) when predicting within each population, genomic prediction of BGEM based on an AP training set resulted in low prediction accuracies. Optimal training population (OTP) design methods fast and unique representative subset selection (FURS), maximization of connectedness and diversity (MaxCD), and partitioning around medoids (PAM) were adapted for inbreds and, along with the methods mean coefficient of determination (CDmean) and mean prediction error variance (PEVmean), often improved prediction accuracies compared with random training sets of the same size. When applied to the combined population, OTP designs enabled successful prediction of the rest of the exotic-derived population. Our findings highlight the importance of leveraging genotype data in training set design to efficiently incorporate new exotic germplasm into a plant breeding program.

Sparse kernel models provide optimization of training set design for genomic prediction in multiyear wheat breeding data.

The success of genomic selection (GS) in breeding schemes relies on its ability to provide accurate predictions of unobserved lines at early stages. Multigeneration data provides opportunities to increase the training data size and thus, the likelihood of extracting useful information from ancestors to improve prediction accuracy. The genomic best linear unbiased predictions (GBLUPs) are performed by borrowing information through kinship relationships between individuals. Multigeneration data usually becomes heterogeneous with complex family relationship patterns that are increasingly entangled with each generation. Under these conditions, historical data may not be optimal for model training as the accuracy could be compromised. The sparse selection index (SSI) is a method for training set (TRN) optimization, in which training individuals provide predictions to some but not all predicted subjects. We added an additional trimming process to the original SSI (trimmed SSI) to remove less important training individuals for prediction. Using a large multigeneration (8 yr) wheat (Triticum aestivum L.) grain yield dataset (n = 68,836), we found increases in accuracy as more years are included in the TRN, with improvements of ∼0.05 in the GBLUP accuracy when using 5 yr of historical data relative to when using only 1 yr. The SSI method showed a small gain over the GBLUP accuracy but with an important reduction on the TRN size. These reduced TRNs were formed with a similar number of subjects from each training generation. Our results suggest that the SSI provides a more stable ranking of genotypes than the GBLUP as the TRN becomes larger.

Genomic prediction of hybrid performance: comparison of the efficiency of factorial and tester designs used as training sets in a multiparental connected reciprocal design for maize silage.

Calibrating a genomic selection model on a sparse factorial design rather than on tester designs is advantageous for some traits, and equivalent for others. In maize breeding, the selection of the candidate inbred lines is based on topcross evaluations using a limited number of testers. Then, a subset of single-crosses between these selected lines is evaluated to identify the best hybrid combinations. Genomic selection enables the prediction of all possible single-crosses between candidate lines but raises the question of defining the best training set design. Previous simulation results have shown the potential of using a sparse factorial design instead of tester designs as the training set. To validate this result, a 363 hybrid factorial design was obtained by crossing 90 dent and flint inbred lines from six segregating families. Two tester designs were also obtained by crossing the same inbred lines to two testers of the opposite group. These designs were evaluated for silage in eight environments and used to predict independent performances of a 951 hybrid factorial design. At a same number of hybrids and lines, the factorial design was as efficient as the tester designs, and, for some traits, outperformed them. All available designs were used as both training and validation set to evaluate their efficiency. When the objective was to predict single-crosses between untested lines, we showed an advantage of increasing the number of lines involved in the training set, by (1) allocating each of them to a different tester for the tester design, or (2) reducing the number of hybrids per line for the factorial design. Our results confirm the potential of sparse factorial designs for genomic hybrid breeding.

LIM Tracker: a software package for cell tracking and analysis with advanced interactivity

Cell tracking is one of the most critical tools for time-lapse image analysis to observe cell behavior and cell lineages over a long period of time. However, the accompanying graphical user interfaces are often difficult to use and do not incorporate seamless manual correction, data analysis tools, or simple training set design tools if it is machine learning based. In this paper, we introduce our cell tracking software “LIM Tracker”. This software has a conventional tracking function consisting of recognition processing and link processing, a sequential search-type tracking function based on pattern matching, and a manual tracking function. LIM Tracker enables the seamless use of these functions. In addition, the system incorporates a highly interactive and interlocking data visualization method, which displays analysis result in real time, making it possible to flexibly correct the data and reduce the burden of tracking work. Moreover, recognition functions with deep learning (DL) are also available, which can be used for a wide range of targets including stain-free images. LIM Tracker allows researchers to track living objects with good usability and high versatility for various targets. We present a tracking case study based on fluorescence microscopy images (NRK-52E/EKAREV-NLS cells or MCF-10A/H2B-iRFP-P2A-mScarlet-I-hGem-P2A-PIP-NLS-mNeonGreen cells) and phase contrast microscopy images (Glioblastoma-astrocytoma U373 cells). LIM Tracker is implemented as a plugin for ImageJ/Fiji. The software can be downloaded from https://github.com/LIMT34/LIM-Tracker.

Robustness of neural network emulations of radiative transfer parameterizations in a state-of-the-art general circulation model

Abstract. The ability of machine-learning-based (ML-based) model components to generalize to the previously unseen inputs and its impact on the stability of the models that use these components have been receiving a lot of recent attention, especially in the context of ML-based parameterizations. At the same time, ML-based emulators of existing physically based parameterizations can be stable, accurate, and fast when used in the model they were specifically designed for. In this work we show that shallow-neural-network-based emulators of radiative transfer parameterizations developed almost a decade ago for a state-of-the-art general circulation model (GCM) are robust with respect to the substantial structural and parametric change in the host model: when used in two 7-month-long experiments with a new GCM, they remain stable and generate realistic output. We concentrate on the stability aspect of the emulators' performance and discuss features of neural network architecture and training set design potentially contributing to the robustness of ML-based model components.

Informed training set design enables efficient machine learning-assisted directed protein evolution.

Directed evolution of proteins often involves a greedy optimization in which the mutation in the highest-fitness variant identified in each round of single-site mutagenesis is fixed. The efficiency of such a single-step greedy walk depends on the order in which beneficial mutations are identified-the process is path dependent. Here, we investigate and optimize a path-independent machine learning-assisted directed evolution (MLDE) protocol that allows in silico screening of full combinatorial libraries. In particular, we evaluate the importance of different protein encoding strategies, training procedures, models, and training set design strategies on MLDE outcome, finding the most important consideration to be the implementation of strategies that reduce inclusion of minimally informative "holes" (protein variants with zero or extremely low fitness) in training data. When applied to an epistatic, hole-filled, four-site combinatorial fitness landscape, our optimized protocol achieved the global fitness maximum up to 81-fold more frequently than single-step greedy optimization. A record of this paper's transparent peer review process is included in the supplemental information.

Training set design in genomic prediction with multiple biparental families.

Genomic selection is a powerful tool to reduce the cycle length and enhance the genetic gain of complex traits in plant breeding. However, questions remain about the optimum design and composition of the training set. In this study, we used 944 soybean [Glycine max (L.) Merr.] recombinant inbred lines from eight families derived through a partial-diallel mating design among five parental lines. The cross-validated prediction accuracies for the six traits seed yield, 1,000-seed weight, protein yield, plant height, protein content, and oil content were high, ranging from 0.79 to 0.87. We investigated among-family predictions, making use of the special mating design with different degrees of relatedness among families. Generally, the prediction accuracy decreased from full-sibs to half-sib families to unrelated families. However, half-sib and unrelated families also showed substantial variation in their prediction accuracy for a given family, which appeared to be caused at least in part by the shared segregation of quantitative trait loci in both the training and prediction sets. Combining several half-sib families in composite training sets generally led to an increase in the prediction accuracy compared with the best family alone. The prediction accuracy increased with the size of the training set, but for comparable prediction accuracy, substantially more half-sibs were required than full-sibs. Collectively, our results highlight the potential of genomic selection for soybean breeding and, in a broader context, illustrate the importance of the targeted design of the training set.

CV-α: designing validations sets to increase the precision and enable multiple comparison tests in genomic prediction

Usually, the comparison among genomic prediction models is based on validation schemes as Repeated Random Subsampling (RRS) or K-fold cross-validation. Nevertheless, the design of training and validation sets has a high effect on the way and subjectiveness we compare models. Those procedures cited above have an overlap across replicates that might cause an overestimated estimate and lack of residuals independence due to resampling issues and might cause less accurate results. Furthermore, ANOVA and multiple-comparison tests, such as Tukey, are not recommended due to assumptions unfulfilled regarding residuals' independence. Thus, we propose a new way to sample observations to build training and validation sets based on cross-validation alpha-based design (CV-α). The CV-α was meant to create several validation scenarios (replicates x folds), regardless of the number of genotypes. Using CV-α, the number of genotypes in the same fold across replicates was much lower than K-fold cross-validation, indicating higher residual independence. Therefore, based on the CV-α results, as proof of concept, via ANOVA, we could compare the proposed methodology to RRS and K-fold cross-validation, applying four genomic prediction models with a simulated and real dataset. Concerning the predictive ability and bias, all validation methods showed similar performance. However, regarding the mean squared error and coefficient of variation, the CV-α method presented the best performance under the evaluated scenarios. Moreover, as it has no additional cost or complexity, it is more reliable and allows non-subjective methods to compare models and factors. Therefore, CV-α can be considered a more precise validation methodology for model selection.

Maximizing efficiency of genomic selection in CIMMYT\u2019s tropical maize breeding program

Key messageHistorical data from breeding programs can be efficiently used to improve genomic selection accuracy, especially when the training set is optimized to subset individuals most informative of the target testing set.The current strategy for large-scale implementation of genomic selection (GS) at the International Maize and Wheat Improvement Center (CIMMYT) global maize breeding program has been to train models using information from full-sibs in a “test-half-predict-half approach.” Although effective, this approach has limitations, as it requires large full-sib populations and limits the ability to shorten variety testing and breeding cycle times. The primary objective of this study was to identify optimal experimental and training set designs to maximize prediction accuracy of GS in CIMMYT’s maize breeding programs. Training set (TS) design strategies were evaluated to determine the most efficient use of phenotypic data collected on relatives for genomic prediction (GP) using datasets containing 849 (DS1) and 1389 (DS2) DH-lines evaluated as testcrosses in 2017 and 2018, respectively. Our results show there is merit in the use of multiple bi-parental populations as TS when selected using algorithms to maximize relatedness between the training and prediction sets. In a breeding program where relevant past breeding information is not readily available, the phenotyping expenditure can be spread across connected bi-parental populations by phenotyping only a small number of lines from each population. This significantly improves prediction accuracy compared to within-population prediction, especially when the TS for within full-sib prediction is small. Finally, we demonstrate that prediction accuracy in either sparse testing or “test-half-predict-half” can further be improved by optimizing which lines are planted for phenotyping and which lines are to be only genotyped for advancement based on GP.

Training set design for machine learning techniques applied to the approximation of computationally intensive first-principles kinetic models

We propose a design procedure for the generation of the training set for Machine Learning algorithms with a specific focus on the approximation of computationally-intensive first-principles kinetic models in catalysis. The procedure is based on the function topology and behavior, by means of the calculation of the discrete gradient, and on the relative importance of the independent variables. We apply the proposed methodology to the tabulation and regression of mean-field and kinetic Monte Carlo models aiming at their coupling with reactor simulations. Our tests – in the context both of mean-field kinetics and kinetic Monte Carlo simulations – show that the procedure is able to design a dataset that requires between 60 and 80% fewer data points to achieve the same approximation accuracy than the one obtained with an evenly distributed grid. This strong reduction in the number of points results in a significant computational gain and a concomitant boost of the approximation efficiency. The Machine Learning algorithms trained with the results of the procedure are then included in both macroscopic reactor models and computational fluid dynamics (CFD) simulations. First, a Plug Flow Reactor is employed to carry out a direct comparison with the solution of the full first-principles kinetic model. The results show an excellent agreement within 0.2% between the models. Then, the CFD simulation of complex tridimensional geometry is carried out by using a tabulated kMC model for CO oxidation on Ruthenium oxide, thus providing a showcase of the capability of the approach in making possible the multiscale simulation of complex chemical reactors.

Evaluation of different virtual screening strategies on the basis of compound sets with characteristic core distributions and dissimilarity relationships.

In this work, computational compound screening strategies on the basis of two- and three-dimensional (2D and 3D) molecular representations were investigated including similarity searching and support vector machine (SVM) ranking. Calculations based on topological fingerprints and molecular shape queries and features were compared. A unique aspect of the analysis setting apart from previous comparisons of 2D and 3D virtual screening approaches has been the design of compound reference, training, and test data sets with controlled incremental increases in intra-set structural diversity and different categories of structural relationships between reference/training and test sets. The use of these data sets made it possible to assess the relative performance of 2D and 3D screening strategies under increasingly challenging conditions ultimately leading to the use of training and test sets with essentially unrelated structures. The results showed that 3D similarity searching had little advantage over 2D searching in identifying active compounds with remote structural relationships. However, 3D SVM models trained on the basis of shape features were superior to other approaches (including 2D SVM) when the detection of structure-activity relationships became increasingly challenging. Such 3D SVM methods has thus far only been little investigated in virtual screening, proving a wealth of opportunities for further analyses.

Improved learning from data competitions through strategic design of training and test data sets

Leveraging the depth and breadth of solutions generated through crowdsourcing can be a powerful accelerator to method development for high consequence problems. While data competitions have become quite popular and prevalent, particularly in supervised learning formats, their implementations by the host are highly variable. Without careful planning, a supervised learning competition is vulnerable to overfitting, where the winning solutions are so closely tuned to the particular set of provided data that they cannot generalize to the general underlying problem of interest to the host. This article outlines important considerations for strategically designing relevant and informative data sets to maximize the learning outcome from hosting a competition. These include: 1. precisely defining the scope of the problem, 2. encouraging participation by competitors from diverse technical backgrounds, 3. specifying the most interesting solution space in order to encourage improvement and distinguish between competitors, 4. strategically generating data sets that enable testing for interpolation and extrapolation to new scenarios of interest, 5. leveraging design of experiment principles for strategic data design while preventing unintentional artifacts in the competition data sets that competitors could exploit without addressing the real problem of interest, and 6. carefully designing the leaderboard scoring metric to select top solutions that closely match the overall competition goals. The methods are illustrated with a recently completed competition in the context of urban radiological search to evaluate algorithms capable of detecting, identifying, and locating radioactive materials. Simulated data were used in the urban search competition. Ideas for using measured real data in competitions are also suggested.

Feature parameter extraction and intelligent estimation of the State-of-Health of lithium-ion batteries

In order to provide an accurate State-Of-Health (SOH) estimation, a novel estimation method is proposed in this paper. In this work, some battery SOH relate features are selected theoretically, proved and then re-screened mathematically. These features can reflect the battery degeneration from different aspects. Also, a new training set design idea is proposed for Least Squares Support Vector Machine algorithm, thereby a model that is suitable for lithium-ion Battery SOH estimation under multi-working conditions can be built. Several lithium-ion battery degeneration testing datasets from National Aeronautics and Space Administration Ames Prognostics Center of Excellence are used to validate the proposed method. Results demonstrate both the superiority of the proposed method and its potential applicability as an effective SOH estimation method for embedded Battery Management System.

The effects of training population design on genomic prediction accuracy in wheat

Genomic selection offers several routes for increasing the genetic gain or efficiency of plant breeding programmes. In various species of livestock, there is empirical evidence of increased rates of genetic gain from the use of genomic selection to target different aspects of the breeder’s equation. Accurate predictions of genomic breeding value are central to this, and the design of training sets is in turn central to achieving sufficient levels of accuracy. In summary, small numbers of close relatives and very large numbers of distant relatives are expected to enable predictions with higher accuracy. To quantify the effect of some of the properties of training sets on the accuracy of genomic selection in crops, we performed an extensive field-based winter wheat trial. In summary, this trial involved the construction of 44 F2:4 bi- and tri-parental populations, from which 2992 lines were grown on four field locations and yield was measured. For each line, genotype data were generated for 25 K segregating SNP markers. The overall heritability of yield was estimated to 0.65, and estimates within individual families ranged between 0.10 and 0.85. Genomic prediction accuracies of yield BLUEs were 0.125–0.127 using two different cross-validation approaches and generally increased with training set size. Using related crosses in training and validation sets generally resulted in higher prediction accuracies than using unrelated crosses. The results of this study emphasise the importance of the training panel design in relation to the genetic material to which the resulting prediction model is to be applied.

Quality monitoring in petroleum refinery with regression neural network: Improving prediction accuracy with appropriate design of training set

The objective of this research is twofold. First, design of training set from the available plant data which is followed by use of training set for developing data driven linear and non-linear soft sensor models for continuous quality monitoring in petroleum refinery. Three data sets from three different processes in the petroleum refinery were investigated. The three data sets belong to ethane-ethylene distillation, debutanization and sulphur recovery process. Five different training set design techniques were applied separately to the three process datasets. These include Kennard-Stone, Duplex, SPXY, KSPXY and SPXYE techniques. Different sets of training data and validation data are designed for the three processes using the five techniques. The resulting training set data are used to develop linear (Multiple Linear Regression) and non-linear (Regression Neural Network) models of the three processes. The resulting validation set data are used to test the generalization ability of the developed models. Subsequently, the function computation time for all five techniques on the three process datasets were determined. It was observed that the duplex technique resulted in the best representative training set. However, the training sets designed from Kennard-Stone and SPXYE techniques resulted in models with best prediction performance with unknown data. The regression neural network models developed from the training set obtained by using Kennard-Stone algorithm for the debutanizer column and sulphur recovery unit are also found to perform better than some other data driven models reported in the literature.