Challenges In Machine Learning Research Articles

The Tracking Machine Learning Challenge: Throughput Phase

This paper reports on the second “Throughput” phase of the Tracking Machine Learning (TrackML) challenge on the Codalab platform. As in the first “Accuracy” phase, the participants had to solve a difficult experimental problem linked to tracking accurately the trajectory of particles as e.g. created at the Large Hadron Collider (LHC): given O(10^5) points, the participants had to connect them into O(10^4) individual groups that represent the particle trajectories which are approximated helical. While in the first phase only the accuracy mattered, the goal of this second phase was a compromise between the accuracy and the speed of inference. Both were measured on the Codalab platform where the participants had to upload their software. The best three participants had solutions with good accuracy and speed an order of magnitude faster than the state of the art when the challenge was designed. Although the core algorithms were less diverse than in the first phase, a diversity of techniques have been used and are described in this paper. The performance of the algorithms is analysed in depth and lessons derived.

A novel firefly algorithm approach for efficient feature selection with COVID-19 dataset

Feature selection is one of the most important challenges in machine learning and data science. This process is usually performed in the data preprocessing phase, where the data is transformed to a proper format for further operations by machine learning algorithm. Many real-world datasets are highly dimensional with many irrelevant, even redundant features. These kinds of features do not improve classification accuracy and can even shrink down performance of a classifier. The goal of feature selection is to find optimal (or sub-optimal) subset of features that contain relevant information about the dataset from which machine learning algorithms can derive useful conclusions. In this manuscript, a novel version of firefly algorithm (FA) is proposed and adapted for feature selection challenge. Proposed method significantly improves performance of the basic FA, and also outperforms other state-of-the-art metaheuristics for both, benchmark bound-constrained and practical feature selection tasks. Method was first validated on standard unconstrained benchmarks and later it was applied for feature selection by using 21 standard University of California, Irvine (UCL) datasets. Moreover, presented approach was also tested for relatively novel COVID-19 dataset for predicting patients health, and one microcontroller microarray dataset. Results obtained in all practical simulations attest robustness and efficiency of proposed algorithm in terms of convergence, solutions’ quality and classification accuracy. More precisely, the proposed approach obtained the best classification accuracy on 13 out of 21 total datasets, significantly outperforming other competitor methods.

Lessons learned from the 1st Ariel Machine Learning Challenge: Correcting transiting exoplanet light curves for stellar spots

Abstract The last decade has witnessed a rapid growth of the field of exoplanet discovery and characterization. However, several big challenges remain, many of which could be addressed using machine learning methodology. For instance, the most prolific method for detecting exoplanets and inferring several of their characteristics, transit photometry, is very sensitive to the presence of stellar spots. The current practice in the literature is identifying the effects of spots visually and correcting them manually or discarding the affected data. This paper explores a first step towards fully automating the efficient and precise derivation of transit depths from transit light curves in the presence of stellar spots. The primary focus of the paper is to present in detail a diverse arsenal of methods for doing so. The methods and results we present were obtained in the context of the 1st Machine Learning Challenge organized for the European Space Agency’s upcoming Ariel mission. We first present the problem, the simulated Ariel-like data and outline the Challenge while identifying best practices for organizing similar challenges in the future. Finally, we present the solutions obtained by the top five winning teams, provide their code, and discuss their implications. Successful solutions either construct highly non-linear (w.r.t. the raw data) models with minimal pre-processing – deep neural networks and ensemble methods – or amount to obtaining meaningful statistics from the light curves, constructing linear models on which yields comparably good predictive performance.

Barren plateaus from learning scramblers with local cost functions

The existence of barren plateaus has recently revealed new training challenges in quantum machine learning (QML). Uncovering the mechanisms behind barren plateaus is essential in understanding the scope of problems that QML can efficiently tackle. Barren plateaus have recently been shown to exist when learning global properties of random unitaries, which is relevant when learning black hole dynamics. Establishing whether local cost functions can circumvent these barren plateaus is pertinent if we hope to apply QML to quantum many-body systems. We prove a no-go theorem showing that local cost functions encounter barren plateaus in learning random unitary properties.

Scalable Neural Network Algorithms for High Dimensional Data

The boundary for machine learning engineers lately has moved from the restricted data to the algorithms' failure to involve every one of the data in the time permitted. Due of this, scientists are presently worried about the adaptability of machine learning algorithms notwithstanding their exactness. The key to success for many computer vision and machine learning challenges is having big training sets. A few published systematic reviews were taken into account in this topic. Recent systematic reviews may include both more recent and older research on the subject under study. Thus, the publications we examined were all recent. The review utilized information that were gathered somewhere in the range of 2010 and 2021. System: In this paper, we make a modified brain organization to eliminate possible components from extremely high layered datasets. Both a totaled level and an exceptionally fine-grained level of translation are feasible for these highlights. It is basically as easy to grasp non-straight connections as it is a direct relapse. We utilize the method on a dataset for item returns in web based shopping that has 15,555 aspects and 5,659,676 all out exchanges. Result and conclusion: We compare 87 various models to show that our approach not only produces higher predicted accuracy than existing techniques, but is also interpretable. The outcomes show that feature selection is a useful strategy for enhancing scalability. The method is sufficiently abstract to be used with many different analytics datasets

A metalearning approach for Physics-Informed Neural Networks (PINNs): Application to parameterized PDEs

Physics-informed neural networks (PINNs) as a means of discretizing partial differential equations (PDEs) are garnering much attention in the Computational Science and Engineering (CS&E) world. At least two challenges exist for PINNs at present: an understanding of accuracy and convergence characteristics with respect to tunable parameters and identification of optimization strategies that make PINNs as efficient as other computational science tools. The cost of PINNs training remains a major challenge of Physics-informed Machine Learning (PiML) – and, in fact, machine learning (ML) in general. This paper is meant to move towards addressing the latter through the study of PINNs on new tasks, for which parameterized PDEs provides a good testbed application as tasks can be easily defined in this context. Following the ML world, we introduce metalearning of PINNs with application to parameterized PDEs. By introducing metalearning and transfer learning concepts, we can greatly accelerate the PINNs optimization process. We present a survey of model-agnostic metalearning, and then discuss our model-aware metalearning applied to PINNs as well as implementation considerations and algorithmic complexity. We then test our approach on various canonical forward parameterized PDEs that have been presented in the emerging PINNs literature.

Analyzing Physics-Inspired Metaheuristic Algorithms in Feature Selection with K-Nearest-Neighbor

In recent years, feature selection has emerged as a major challenge in machine learning. In this paper, considering the promising performance of metaheuristics on different types of applications, six physics-inspired metaphor algorithms are employed for this problem. To evaluate the capability of dimensionality reduction in these algorithms, six diverse-natured datasets are used. The performance is compared in terms of the average number of features selected (AFS), accuracy, fitness, convergence capabilities, and computational cost. It is found through experiments that the accuracy and fitness of the Equilibrium Optimizer (EO) are comparatively better than the others. Finally, the average rank from the perspective of average fitness, average accuracy, and AFS shows that EO outperforms all other algorithms.

Multi-label Classification Using Vector Generalized Additive Model via Cross-Validation

Multi-label classification is a unique challenge in machine learning designed for two targets with each containing one or multiple classes. This problem can be resolved using several methods, including the classification of the targets individually or simultaneously. However, most models cannot classify the target simultaneously, and this is not expected to happen in the modeling rule. This study was conducted to propose a novel solution in the form of a Vector Generalized Additive Model Using Cross-Validation (VGAMCV) to address these problems. The proposed method leverages the Vector Generalized Additive Model (VGAM), which is a semi-parametric model combining both parametric and non-parametric components as the underlying base model. Cross-validation was also applied to tune the parameters to optimize the performance of the method. Moreover, the methodology of VGAMCV was compared with a tree-based model, Random Forest, commonly used in multi-label classification to evaluate its effectiveness based on fourteen metric scores. The results showed positive outcomes as indicated by 0.703 average accuracy and 0.601 Area Under Curve (AUC) recorded, but these improvements were not statistically significant. Meanwhile, the method offered a viable alternative for multi-label classification tasks, and its introduction served as a contribution to the expanding repertoire of methods available for this purpose.

Generating Natural Language From Logic Expressions With Structural Representation

Incorporating logic reasoning with deep neural networks (DNNs) is an important challenge in machine learning. In this article, we study the problem of converting logical expressions into natural language. In particular, given a sequential logic expression, the goal is to generate its corresponding natural sentence. Since the information in a logic expression often has a hierarchical structure, a sequence-to-sequence baseline struggles to capture the full dependencies between words, and hence it often generates incorrect sentences. To alleviate this problem, we propose a model to convert Structural Logic Expressions into Natural Language (SLEtoNL). SLEtoNL converts sequential logic expressions into structural representation and leverages structural encoders to capture the dependencies between nodes. The quantitative and qualitative analyses demonstrate that our proposed method outperforms the seq2seq model, which is based on the sequential representation, and outperforms strong pretrained language models ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , T5, BART, GPT3) with a large margin (28.6 in BLEU3) in out-of-distribution evaluation. The data and code will be available on <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com</uri> .

Feature selection for classification using WGCNA and Spread Sub-Sample for an imbalanced rheumatoid arthritis RNASEQ data

An imbalanced classification problem occurs when the distribution of samples among different classes is uneven or biased. Handling small and imbalanced training datasets poses a notable challenge in machine learning, especially in domains such as bioinformatics and medical research. These challenges can result in biased models, leading to poor performance on under-represented classes and an overemphasis on specific features, failing to capture the genuine patterns present in the data. The present study proposes a feature selection approach-based on genes connectivity and a class balancing technique for building a machine leaning model using imbalanced gene expression data. Rheumatic arthritis data composed of 28 normal samples and 152 rheumatic samples was used in testing our proposed model. Through the weighted gene co-expression network analysis (WGCNA) approach, features were reduced to 601 from 27,991 original features. The reduced features were used to build machine learning classification models with imbalanced and later balanced classes using Spread Sub-Sample technique. According to our findings, two classifiers reported higher accuracy with imbalanced data as compared to the balanced data set. This is an indication that most classifiers are biased when trained using imbalanced dataset. Logistic regression returned improved accuracy of 95%. The other two machine learning algorithms used in this study were decision tree and IBK returned reduced accuracy of 81% and 91% respectively. In conclusion, feature selection and class balancing approaches are important in reducing model execution time and accuracy especially for RNASeq gene expression data.

Learning From Imbalanced Data Using Triplet Adversarial Samples

The imbalance of classes in real-world datasets poses a major challenge in machine learning and classification, and traditional synthetic data generation methods often fail to address this problem effectively. A major limitation of these methods is that they tend to separate the process of generating synthetic samples from the training process, resulting in synthetic data that lack the necessary informative characteristics for proper model training. We present a new synthetic data generation method that addresses this issue by combining adversarial sample generation with a triplet loss method. This approach focuses on increasing the diversity in the minority class while preserving the integrity of the decision boundary. Furthermore, we show that reducing triplet loss is equivalent to maximizing the area under the receiver operating characteristic curve under specific conditions, providing a theoretical basis for the effectiveness of our method. In addition, we present a model training approach to further improve the generalization of the model to small classes by providing a diverse set of synthetic samples optimized using our proposed loss function. We evaluated our method on several imbalanced benchmark tasks and compared it to state-of-the-art techniques, demonstrating that our method can deliver even better performance, making it an effective solution to the class imbalance problem.

A Heuristic Machine Learning-Based Optimization Technique to Predict Lung Cancer Patient Survival.

Cancer has been a significant threat to human health and well-being, posing the biggest obstacle in the history of human sickness. The high death rate in cancer patients is primarily due to the complexity of the disease and the wide range of clinical outcomes. Increasing the accuracy of the prediction is equally crucial as predicting the survival rate of cancer patients, which has become a key issue of cancer research. Many models have been suggested at the moment. However, most of them simply use single genetic data or clinical data to construct prediction models for cancer survival. There is a lot of emphasis in present survival studies on determining whether or not a patient will survive five years. The personal issue of how long a lung cancer patient will survive remains unanswered. The proposed technique Naive Bayes and SSA is estimating the overall survival time with lung cancer. Two machine learning challenges are derived from a single customized query. To begin with, determining whether a patient will survive for more than five years is a simple binary question. The second step is to develop a five-year survival model using regression analysis. When asked to forecast how long a lung cancer patient would survive within five years, the mean absolute error (MAE) of this technique's predictions is accurate within a month. Several biomarker genes have been associated with lung cancers. The accuracy, recall, and precision achieved from this algorithm are 98.78%, 98.4%, and 98.6%, respectively.

Models developed for spiking neural networks

Emergence of deep neural networks (DNNs) has raised enormous attention towards artificial neural networks (ANNs) once again. They have become the state-of-the-art models and have won different machine learning challenges. Although these networks are inspired by the brain, they lack biological plausibility, and they have structural differences compared to the brain. Spiking neural networks (SNNs) have been around for a long time, and they have been investigated to understand the dynamics of the brain. However, their application in real-world and complicated machine learning tasks were limited. Recently, they have shown great potential in solving such tasks. Due to their energy efficiency and temporal dynamics there are many promises in their future development. In this work, we reviewed the structures and performances of SNNs on image classification tasks. The comparisons illustrate that these networks show great capabilities for more complicated problems. Furthermore, the simple learning rules developed for SNNs, such as STDP and R-STDP, can be a potential alternative to replace the backpropagation algorithm used in DNNs.•Different building blocks of spiking neural networks are explained in this work.•Developed models for SNNs are introduced based on their characteristics and building blocks.

Two Novel SMOTE Methods for Solving Imbalanced Classification Problems

The imbalanced classification problem has always been one of the important challenges in neural network and machine learning. As an effective method to deal with imbalanced classification problems, the synthetic minority oversampling technique (SMOTE) has its disadvantage: Some noise samples may participate in the process of synthesizing new samples; As a result, the new synthetic sample lacks its rationality, which will reduce the classification performances of the network. To remedy this shortcoming, two novel improved SMOTE method are proposed in this paper: Center point SMOTE (CP-SMOTE) method and Inner and outer SMOTE (IO-SMOTE) method. The CP-SMOTE method generates new samples based on finding several center points, then linearly combining the minority samples with their corresponding center points. The IO-SMOTE method divides minority samples into inner and outer samples, and then uses inner samples as much as possible in the subsequent process of generating new samples. Numerical experiments are conducted to prove that compared with no-sampling and conventional SMOTE methods, the CP-SMOTE and IO-SMOTE methods can achieve better classification performances.

Applying neural networks to analyse the properties and structure of composite materials

In this paper, we show how a simple convolutional neural network (CNN) can be trained to predict homogenised elastic properties of a composite material based on its representative volume element (RVE). We consider both 2D and 3D composites featuring two components with fixed isotropic elastic properties. The dataset used to train the neural network is obtained in two stages. The first stage is the generation of random RVEs and computation of their homogenised elastic properties using finite element analysis (FEA) with periodic boundary conditions (PBC) applied to them, and the second stage is the application of a data augmentation scheme to that dataset obtained in the first stage. For the 2D and 3D cases, we present two neural networks and show that they are able to estimate the values of homogenised elastic properties fairly accurately. At the same time, we confirm that the data generation stage is computationally very expensive and is a major challenge for machine learning based techniques in computational mechanics. Indeed, it is expensive even for “low resolution” RVEs such as those considered in our work, i.e. squares (2D case) or cubes (3D case) uniformly subdivided into 8x8 (2D case) or 8x8x8 (3D case) domains. We discuss the flaws and drawbacks as well as the possible developments and uses of this approach in multiscale modeling of composite materials.

Personalized Federated Learning for In-Hospital Mortality Prediction of Multi-Center ICU

Federated learning (FL), as a paradigm for addressing challenges of machine learning (ML) to be applied in private distributed data provides a novel and promising scheme to promote ML in multiple independently distributed healthcare institutions. However, the non-IID and unbalanced nature of the data distribution can decrease its performance, even resulting in the institutions losing motivation to participate in its training. This paper explored the problem with an in-hospital mortality prediction task under an actual multi-center ICU electronic health record database that preserves the original non-IID and unbalanced data distribution. It first analyzed the reason for the performance degradation of baseline FL under this data scenario, and then proposed a personalized FL (PFL) approach named POLA to tackle the problem. POLA is a personalized one-shot and two-step FL method capable of generating high-performance personalized models for each independent participant. The proposed method, POLA was compared with two other PFL methods in experiments, and the results indicate that it not only effectively improves the prediction performance of FL but also significantly reduces the communication rounds. Moreover, its generality and extensibility also make it potential to be extended to other similar cross-silo FL application scenarios.

The 10 most popular Concept Drift Algorithms: An overview and optimization potentials

In a dynamic world, data streams are continuously generated, which poses immense challenges for machine learning (ML) algorithms to adapt to changing statistical properties that are subject to a non-stationary context. The underlying scenario is defined as concept drift (CD), where changes in the relationship between response and prediction variables (real CD) or a change in input data (virtual CD) are accompanied by a significant degradation in the predictive performance of the models, causing ML models to reach unacceptable levels of system accuracy. In this paper, the state of the art for CD algorithms is analyzed and compared. For this purpose, a systematic literature review was performed. Then, the 10 most popular CD algorithms were extracted from the literature using a newly-developed metric. Subsequently, the algorithms were analyzed and compared with respect to their functionality and limitations. Based on these, the optimization potentials were systematically derived. This work presents a summarized overview of CD algorithms and provides the basis for algorithm optimization in this domain.

Investigating Group Distributionally Robust Optimization for Deep Imbalanced Learning: A Case Study of Binary Tabular Data Classification

One of the most studied machine learning challenges that recent studies have shown the susceptibility of deep neural networks to is the class imbalance problem. While concerted research efforts in this direction have been notable in recent years, findings have shown that the canonical learning objective, empirical risk minimization (ERM), is unable to achieve optimal imbalance learning in deep neural networks given its bias to the majority class. An alternative learning objective, group distributionally robust optimization (gDRO), is investigated in this study for imbalance learning, focusing on tabular imbalanced data as against image data that has dominated deep imbalance learning research. Contrary to minimizing average per instance loss as in ERM, gDRO seeks to minimize the worst group loss over the training data. Experimental findings in comparison with ERM and classical imbalance methods using four popularly used evaluation metrics in imbalance learning across several benchmark imbalance binary tabular data of varying imbalance ratios reveal impressive performance of gDRO, outperforming other compared methods in terms of g-mean and roc-auc.

A bio-inspired weights and structure determination neural network for multiclass classification: Applications in occupational classification systems

&lt;abstract&gt;&lt;p&gt;Undoubtedly, one of the most common machine learning challenges is multiclass classification. In light of this, a novel bio-inspired neural network (NN) has been developed to address multiclass classification-related issues. Given that weights and structure determination (WASD) NNs have been acknowledged to alleviate the disadvantages of conventional back-propagation NNs, such as slow training pace and trapping in a local minimum, we developed a bio-inspired WASD algorithm for multiclass classification problems (BWASDC) by using the metaheuristic beetle antennae search (BAS) algorithm to enhance the WASD algorithm's learning process. The BWASDC's effectiveness is then evaluated through applications in occupational classification systems. It is important to mention that systems of occupational classification serve as a fundamental indicator of occupational exposure. For this reason, they are highly significant in social science research. According to the findings of four occupational classification experiments, the BWASDC model outperformed some of the most modern classification models obtainable through MATLAB's classification learner app on all fronts.&lt;/p&gt;&lt;/abstract&gt;

Opportunities and challenges of machine learning in bioprocesses: Categorization from different perspectives and future direction

Recent advances in machine learning (ML) have revolutionized an extensive range of research and industry fields by successfully addressing intricate problems that cannot be resolved with conventional approaches. However, low interpretability and incompatibility make it challenging to apply ML to complicated bioprocesses, which rely on the delicate metabolic interplay among living cells. This overview attempts to delineate ML applications to bioprocess from different perspectives, and their inherent limitations (i.e., uncertainties in prediction) were then discussed with unique attempts to supplement the ML models. A clear classification can be made depending on the purpose of the ML (supervised vs unsupervised) per application, as well as on their system boundaries (engineered vs natural). Although a limited number of hybrid approaches with meaningful outcomes (e.g., improved accuracy) are available, there is still a need to further enhance the interpretability, compatibility, and user-friendliness of ML models.