Regression Problems Research Articles

Genetic Programming With Lexicase Selection for Large-Scale Dynamic Flexible Job Shop Scheduling

Dynamic flexible job shop scheduling is a prominent combinatorial optimisation problem with many real-world applications. Genetic programming has been widely used to automatically evolve effective scheduling heuristics for dynamic flexible job shop scheduling. A limitation of genetic programming is the premature convergence due to the loss of population diversity. To overcome this limitation, this work considers using lexicase selection to improve population diversity, which has achieved success on regression and program synthesis problems. However, it is not trivial to apply lexicase selection to genetic programming for dynamic flexible job shop scheduling, since a fitness case (training scheduling simulation) is often large-scale, making the fitness evaluation very time-consuming. To address this issue, we propose a new multi-case fitness scheme, which creates multiple cases from a single scheduling simulation. Based on the multi-case fitness, we develop a new genetic programming algorithm with lexicase selection, which uses a single simulation for fitness evaluation, thus achieving a better balance between the number of cases for lexicase selection and evaluation efficiency. The experiments on a wide range of dynamic scheduling scenarios show that the proposed algorithm can achieve better population diversity and final performance than the current genetic programming parent selection methods and a state-of-the-art deep reinforcement learning method.

TC-DTA: Predicting Drug-Target Binding Affinity With Transformer and Convolutional Neural Networks.

Bioinformatics is a rapidly evolving field that applies computational methods to analyze and interpret biological data. A key task in bioinformatics is identifying novel drug-target interactions (DTIs), which plays a crucial role in drug discovery. Most computational approaches treat DTI prediction as a binary classification problem, determining whether drug-target pairs interact. However, with the growing availability of drug-target binding affinity data, this binary task can be reframed as a regression problem focused on drug-target affinity (DTA). DTA quantifies the strength of drug-target binding, offering more detailed insights than DTI and serving as a valuable tool for virtual screening in drug discovery. Accurately predicting compound interactions with targets can accelerate the drug development process. In this study, we introduce a deep learning model named TC-DTA for DTA prediction, leveraging convolutional neural networks (CNN) and the encoder module of the transformer architecture. We begin by extracting raw drug SMILES strings and protein amino acid sequences from the dataset, which are then represented using various encoding methods. Subsequently, we employ CNN and the transformer's encoder module to extract features from the drug SMILES strings and protein sequences, respectively. Finally, the feature information is concatenated and input into a multi-layer perceptron to predict binding affinity scores. We evaluated our model on two benchmark DTA datasets, Davis and KIBA, comparing it with methods such as KronRLS, SimBoost, and DeepDTA. Our model, TC-DTA, outperformed these baseline methods based on evaluation metrics like Mean Squared Error (MSE), Concordance Index (CI), and Regression towards the Mean Index ( rm2 ). These results highlight the effectiveness of the Transformer's encoder and CNN in extracting meaningful representations from sequences, thereby enhancing DTA prediction accuracy. This deep learning model can accelerate drug discovery by identifying drug candidates with high binding affinity to specific targets. Compared to traditional methods, machine learning technology offers a more effective and efficient approach to drug discovery.

Least squares regression under weak moment conditions

In this paper we consider the robust regression problem when the output variable may be heavy-tailed. In such scenarios, the traditional least squares regression paradigm is usually thought to be not a good choice as it lacks robustness to outliers. By projecting the outputs onto an adaptive interval, we show the regularized least squares regression can still work well when the conditional distribution satisfies a weak moment condition. Fast convergence rates in various norm are derived by tuning the projection scale parameter and regularization parameter in according with the sample size and the moment condition.

Differential evolution with adaptive mutation and crossover strategies for nonlinear regression problems

Differential evolution with adaptive mutation and crossover strategies for nonlinear regression problems

Multi-model subset selection

The two primary approaches for high-dimensional regression problems are sparse methods (e.g., best subset selection, which uses the ℓ0-norm in the penalty) and ensemble methods (e.g., random forests). Although sparse methods typically yield interpretable models, in terms of prediction accuracy they are often outperformed by “blackbox” multi-model ensemble methods. A regression ensemble is introduced which combines the interpretability of sparse methods with the high prediction accuracy of ensemble methods. An algorithm is proposed to solve the joint optimization of the corresponding ℓ0-penalized regression models by extending recent developments in ℓ0-optimization for sparse methods to multi-model regression ensembles. The sparse and diverse models in the ensemble are learned simultaneously from the data. Each of these models provides an explanation for the relationship between a subset of predictors and the response variable. Empirical studies and theoretical knowledge about ensembles are used to gain insight into the ensemble method's performance, focusing on the interplay between bias, variance, covariance, and variable selection. In prediction tasks, the ensembles can outperform state-of-the-art competitors on both simulated and real data. Forward stepwise regression is also generalized to multi-model regression ensembles and used to obtain an initial solution for the algorithm. The optimization algorithms are implemented in publicly available software packages.

Formation pressure prediction method based on machine learning

Abstract Formation pore pressure is one of the key parameters in petroleum engineering. The high-temperature and high-pressure working conditions formed under the complex mechanism of Yingqiong Basin make downhole complex events such as overflow and well leakage occur frequently. To address the problem of the large prediction error of the traditional Eaton method under high temperature and high-pressure conditions, three machine learning methods, such as GA-BP(Genetic Algorithm-Back Propagation Neural Network), PSO-LSTM(Particle Swarm Optimization-Long Short-Term Memory Neural Network), SVR(Support Vector Regression), are introduced to find the nonlinear relationship between logging parameters and formation pore pressure. Seven logging parameters such as formation depth, mechanical drilling speed, drilling pressure, and rotational speed were used as input variables, and formation pore pressure was used as an output variable through correlation analysis for constructing the machine learning model. By analyzing the experimental results of the above three machine learning models with the Eaton method, the GA-BP model with lower error and more satisfying the needs of drilling engineering is preferred, and the various regression evaluation indexes of the model are also more excellent compared with other machine learning models, which are Mean Squared Error (MSE)=0.001, Mean Absolute Error (MAE)=0.003, and R-squared (R2)=0.991, respectively. It is concluded that the machine learning method is not only more advantageous in a class of nonlinear regression problems such as formation pore pressure prediction, but also can help engineers to predict formation pore pressure more accurately under the conditions of deep high-temperature, high-pressure, and complex formations, which contributes to guaranteeing the safety and high efficiency of drilling projects.

An efficient Newton-type matrix splitting algorithm for solving generalized absolute value equations with application to ridge regression problems

A generalized Newton-based matrix splitting (GNMS) method is proposed for solving the generalized absolute value equations (GAVEs). Under mild conditions, the GNMS method converges to the unique solution of GAVEs. Moreover, we can obtain a few weaker convergence conditions for some existing methods. Numerical results verify the effectiveness of the proposed method.

Large language models as surrogate models in evolutionary algorithms: A preliminary study

Large Language Models (LLMs) have demonstrated remarkable advancements across diverse domains, manifesting considerable capabilities in evolutionary computation, notably in generating new solutions and automating algorithm design. Surrogate-assisted selection plays a pivotal role in evolutionary algorithms (EAs), especially in addressing expensive optimization problems by reducing the number of real function evaluations. However, whether LLMs can serve as surrogate models remains an unknown. In this study, we propose a novel surrogate model based purely on LLM inference capabilities, eliminating the need for training. Specifically, we formulate model-assisted selection as a classification problem or a regression problem, utilizing LLMs to directly evaluate the quality of new solutions based on historical data. This involves predicting whether a solution is good or bad, or approximating its value. This approach is then integrated into EAs, termed LLM-assisted EA (LAEA). Detailed experiments compared the visualization results of 2D data from 9 mainstream LLMs, as well as their performance on 5-10 dimensional problems. The experimental results demonstrate that LLMs have significant potential as surrogate models in evolutionary computation, achieving performance comparable to traditional surrogate models only using inference. This work offers new insights into the application of LLMs in evolutionary computation. Code is available at: https://github.com/hhyqhh/LAEA.git.

FAStEN: An Efficient Adaptive Method for Feature Selection and Estimation in High-Dimensional Functional Regressions

Functional regression analysis is an established tool for many contemporary scientific applications. Regression problems involving large and complex data sets are ubiquitous, and feature selection is crucial for avoiding overfitting and achieving accurate predictions. We propose a new, flexible and ultra-efficient approach to perform feature selection in a sparse high dimensional function-on-function regression problem, and we show how to extend it to the scalar-on-function framework. Our method, called FAStEN, combines functional data, optimization, and machine learning techniques to perform feature selection and parameter estimation simultaneously. We exploit the properties of Functional Principal Components and the sparsity inherent to the Dual Augmented Lagrangian problem to significantly reduce computational cost, and we introduce an adaptive scheme to improve selection accuracy. In addition, we derive asymptotic oracle properties, which guarantee estimation and selection consistency for the proposed FAStEN estimator. Through an extensive simulation study, we benchmark our approach to the best existing competitors and demonstrate a massive gain in terms of CPU time and selection performance, without sacrificing the quality of the coefficients’ estimation. The theoretical derivations and the simulation study provide a strong motivation for our approach. Finally, we present an application to brain fMRI data from the AOMIC PIOP1 study. Complete FAStEN code is provided at https://github.com/IBM/funGCN. Supplementary materials for this article are available online.

Ensemble Learning-Based Fuzzy Aggregation Functions and Their Application in TSK Neural Networks

AbstractTakagi–Sugeno–Kang fuzzy neural networks (TSKFNN) are powerful tools to model vague or imprecise information. Regression is one of the most important tasks commonly tackled by supervised learning techniques. TSKFNNs are considered suitable models to deal with regression problems on account of their simplicity and flexibility. Aggregation methods play an important role in combining various fuzzy rules from a TSKFNN rule base to obtain a model prediction. However, many current aggregation methods rely on expert experience and human knowledge, which may be hard to acquire and could bring human bias. This paper proposes data-driven aggregation functions for rules aggregation based on ensemble learning, namely AdaBoost and bagging, which can achieve superior generalizability in testing compared with the existing rule aggregation methods. Furthermore, they can also provide insights into the importance of each rule in the model’s decision making, thus, helping to improve the interpretability of the model. Extensive experiments on 11 commonly used benchmark datasets with various sizes and dimensionalities validated the superiority of the proposed ensemble learning-based fuzzy aggregation functions compared with existing state-of-the-art TSKFNNs.

A Network‐Constrain Weibull AFT Model for Biomarkers Discovery

ABSTRACTWe propose AFTNet, a novel network‐constraint survival analysis method based on the Weibull accelerated failure time (AFT) model solved by a penalized likelihood approach for variable selection and estimation. When using the log‐linear representation, the inference problem becomes a structured sparse regression problem for which we explicitly incorporate the correlation patterns among predictors using a double penalty that promotes both sparsity and grouping effect. Moreover, we establish the theoretical consistency for the AFTNet estimator and present an efficient iterative computational algorithm based on the proximal gradient descent method. Finally, we evaluate AFTNet performance both on synthetic and real data examples.

Nonparametric Multiple-Output Center-Outward Quantile Regression

Building on recent measure-transportation-based concepts of multivariate quantiles, we are considering the problem of nonparametric multiple-output quantile regression. Our approach defines nested conditional center-outward quantile regression contours and regions with given conditional probability content, the graphs of which constitute nested center-outward quantile regression tubes with given unconditional probability content; these (conditional and unconditional) probability contents do not depend on the underlying distribution—an essential property of quantile concepts. Empirical counterparts of these concepts are constructed, yielding interpretable empirical contours, regions, and tubes which are shown to consistently reconstruct (in the Pompeiu-Hausdorff topology) their population versions. Our method is entirely nonparametric and performs well in simulations—with possible heteroscedasticity and nonlinear trends. Its potential as a data-analytic tool is illustrated on some real datasets. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

5 Hands-on 3: Introduction to developing Python computational pipelines for predictive machine learning modelling of livestock data

Abstract This hands-on workshop aims to offer examples of computational pipelines that apply traditional predictive machine learning modelling techniques to animal science datasets with the purpose of solving classification and/or regression problems. The objective of this workshop is to provide the attendants with fully functional and customizable computational pipelines developed in Python. The workshop is structured in a hands-on format and includes a combination of basic notions about machine learning and relevant algorithms, evaluation measures, evaluation strategies and Python code examples. To avoid technical problems related to installing a Python environment on personal computers, we recommend the registrants to acquire access on the Repl.it platform (https://replit.com/) by creating a free account prior to attending the workshop. Detailed information will be provided before the beginning of the workshop at the following URL: http://animalbiosciences.uoguelph.ca/~dtulpan/conferences/asas2024/

Rapid identification of phycobiliproteins in Porphyra yezoensis using near-infrared combined with convolutional neural network

Phycobiliproteins (PBPs) are the main pigment proteins and important indicators for evaluating the quality of Porphyra yezoensis (nori). This study aimed to develop a non-destructive and rapid method for determining PBP content (including total PBPs, phycoerythrin, phycocyanin, and allophycocyanin) in P. yezoensis, using near-infrared spectrum technology combined with a convolutional neural network (CNN), and to explore the influence of spectral preprocessing methods and machine learning algorithms on the predictive ability of the model. First, the spectral data was standardized using a combination of standard normal variable transformation and the first derivative to improve the accuracy and predictive ability of the model. We compared various models and determined that the CNN model performed better than conventional methods. After optimizing the number of convolutional layers, dropout rate, and learning rate, the performance of the CNN model was further improved. This study demonstrates the capability of the CNN model to leverage spectral data and solve regression problems to accurately measure PBPs. Moreover, for the first time, we established a functional equation between PBPs and P. yezoensis grades. This study provides a feasible and rapid method for the quantitative detection of PBPs in P. yezoensis.

Bridging auditory perception and natural language processing with semantically informed deep neural networks

Sound recognition is effortless for humans but poses a significant challenge for artificial hearing systems. Deep neural networks (DNNs), especially convolutional neural networks (CNNs), have recently surpassed traditional machine learning in sound classification. However, current DNNs map sounds to labels using binary categorical variables, neglecting the semantic relations between labels. Cognitive neuroscience research suggests that human listeners exploit such semantic information besides acoustic cues. Hence, our hypothesis is that incorporating semantic information improves DNN’s sound recognition performance, emulating human behaviour. In our approach, sound recognition is framed as a regression problem, with CNNs trained to map spectrograms to continuous semantic representations from NLP models (Word2Vec, BERT, and CLAP text encoder). Two DNN types were trained: semDNN with continuous embeddings and catDNN with categorical labels, both with a dataset extracted from a collection of 388,211 sounds enriched with semantic descriptions. Evaluations across four external datasets, confirmed the superiority of semantic labeling from semDNN compared to catDNN, preserving higher-level relations. Importantly, an analysis of human similarity ratings for natural sounds, showed that semDNN approximated human listener behaviour better than catDNN, other DNNs, and NLP models. Our work contributes to understanding the role of semantics in sound recognition, bridging the gap between artificial systems and human auditory perception.

GC[formula omitted]: Grouped convolutional color constancy

Computational color constancy is an ill-posed problem since different combinations of objects and illumination may appear the same color in an image. Most methods produce a single illuminant estimate and cannot explain the potential ambiguity between possible illuminant solutions. In this work, a grouped convolutional color constancy (GC3) method is proposed. It approaches the computational color constancy as a grouped regression problem, dealing with ambiguity by generating multiple possible illuminant solutions. The GC3 consists of two components: fast feature extraction (FFE) and grouped illuminant regression (GIR). FFE uses small convolution kernels and rapid down-sampling scheme for quick feature extraction, while GIR employs grouped convolution layers for feature grouping regression, which estimates the illuminant solution and its probability for each group with two separate branches. The final illumination estimation is obtained by averaging all illuminant solutions, weighted by the normalized probability for each group. The proposed method yields competitive results on the ColorChecker dataset and achieves the state-of-the-art accuracy on the NUS and Cube+ datasets. Moreover, when compared with the state-of-the-art methods, our proposed approach exhibits superior time efficiency, making it the most efficient method available.

An innovative variant based on generative adversarial network (GAN): Regression GAN combined with hyperspectral imaging to predict pesticide residue content of Hami melon

The rapid and non-destructive detection of pesticide residues in Hami melons plays substantial importance in protecting consumer health. However, the investment of time and resources needed to procure sample data poses a challenge, often resulting in limited data set and consequently leading insufficient accuracy of the established models. In this study, an innovative variant based on generative adversarial network (GAN) was proposed, named regression GAN (RGAN). It was used to synchronically extend the visible near-infrared (VNIR) and short-wave infrared (SWIR) hyperspectral data and corresponding acetamiprid residue content data of Hami melon. The support vector regression (SVR) and partial least squares regression (PLSR) models were trained using the generated data, and subsequently validate them with real data to assess the reliability of the generated data. In addition, the generated data were added to the real data to extend the dataset. The SVR model based on SWIR-HSI data achieved the optimal performance after data augmentation, yielding the values of Rp2, RMSEP and RPD were 0.8781, 0.6962 and 2.7882, respectively. The RGAN extends the range of GAN applications from classification problems to regression problems. It serves as a valuable reference for the quantitative analysis of chemometrics.

On-device training of artificial intelligence models on microcontrollers

<p><span>Numerous studies are currently training artificial intelligence (AI) models on tiny devices constrained by computing power and memory limitations by implementing model optimization algorithms. The question arises whether implementing traditional AI models directly on small devices like micro-controller units (MCUs) is feasible. In this study, a library has been developed to train and predict the artificial neural network (ANN) model on common MCUs. The evaluation results on the regression problem indicate that, despite the extensive training time, when combined with multitasking programming on multi-core MCUs, the training does not adversely affect the system's execution. This research contributes an additional solution that enables the direct construction of ANN models on MCU systems with limited resources.</span></p>

ResGAT: Residual Graph Attention Networks for molecular property prediction

Molecular property prediction is an important step in the drug discovery pipeline. Numerous computational methods have been developed to predict a wide range of molecular properties. While recent approaches have shown promising results, no single architecture can comprehensively address all tasks, making this area persistently challenging and requiring substantial time and effort. Beyond traditional machine learning and deep learning architectures for regular data, several deep learning architectures have been designed for graph-structured data to overcome the limitations of conventional methods. Utilizing graph-structured data in quantitative structure–activity relationship (QSAR) modeling allows models to effectively extract unique features, especially where connectivity information is crucial. In our study, we developed residual graph attention networks (ResGAT), a deep learning architecture for molecular graph-structured data. This architecture is a combination of graph attention networks and shortcut connections to address both regression and classification problems. It is also customizable to adapt to various dataset sizes, enhancing the learning process based on molecular patterns. When tested multiple times with both random and scaffold sampling strategies on nine benchmark molecular datasets, QSAR models developed using ResGAT demonstrated stability and competitive performance compared to state-of-the-art methods.

Restoration of the current transformer secondary current under core saturation conditions based on ANN

Relay protection and emergency automation is an integral part of electric power systems. The algorithms of the relay protection and emergency automation are based on the use of analysis of the parameters of the electrical mode - current and voltage - obtained from current and voltage instrument transformers. As a rule, these transformers operate on the basis of the laws of electromagnetism. Operational experience shows that the magnetic core of current transformers under fault conditions can be saturated. As a result, the shape of the measured current is distorted which can lead to false operation of the relay protection and emergency automation system. In this paper a comparative analysis of the existing methods is presented. The advantages and disadvantages of them are described. The problem of current transformer core saturation is reduced to a regression problem. To solve the problem an artificial neural networks-based method is proposed. Computational experiments for both proposed and existing methods are performed. The results of experiments show that the efficiency of the proposed method is 1.25 times higher relative to the most effective existing method. The method of re-scaling data at real-time work applied to the restoration of the distorted shape of current is proposed.