Machine Learning Theory Research Articles

Understanding Machine Learning Principles: Learning, Inference, Generalization, and Computational Learning Theory

Machine learning has become indispensable across various domains, yet understanding its theoretical underpinnings remains challenging for many practitioners and researchers. Despite the availability of numerous resources, there is a need for a cohesive tutorial that integrates foundational principles with state-of-the-art theories. This paper addresses the fundamental concepts and theories of machine learning, with an emphasis on neural networks, serving as both a foundational exploration and a tutorial. It begins by introducing essential concepts in machine learning, including various learning and inference methods, followed by criterion functions, robust learning, discussions on learning and generalization, model selection, bias–variance trade-off, and the role of neural networks as universal approximators. Subsequently, the paper delves into computational learning theory, with probably approximately correct (PAC) learning theory forming its cornerstone. Key concepts such as the VC-dimension, Rademacher complexity, and empirical risk minimization principle are introduced as tools for establishing generalization error bounds in trained models. The fundamental theorem of learning theory establishes the relationship between PAC learnability, Vapnik–Chervonenkis (VC)-dimension, and the empirical risk minimization principle. Additionally, the paper discusses the no-free-lunch theorem, another pivotal result in computational learning theory. By laying a rigorous theoretical foundation, this paper provides a comprehensive tutorial for understanding the principles underpinning machine learning.

Enhancing Coffee Leaf Rust Detection Using DenseNet201: A Comprehensive Analysis of the Mbozi and Public Datasets in Songwe, Tanzania

Coffee Leaf Rust (CLR) is a worldwide devastating fungal disease that threatens coffee production, upsetting economic and farmers' livelihoods. Traditional methods of detecting CLR heavily rely on using machine-learning (ML) models trained through weakly collected datasets and physical inspection which is tedious, time-consuming, and subject to human error. This study explores the performance of the DenseNet201 model using three datasets: Mbozi, Public, and Combined (a merger of Mbozi and Public datasets). Machine Learning Theory guided this research. The study objective is to assess the influence of dataset quality on CLR detection, analyze Mbozi and Public datasets using DenseNet201, and enhance robustness by merging the two datasets. A study on coffee leaf rot (CLR) severity was conducted using systematic sampling techniques. Leaves from multiple coffee farms were collected, representing different levels of infection. The Mbozi dataset, sourced from high-resolution images captured from Tanzania's Songwe coffee plantations, was analyzed for quality under controlled conditions, including environmental factors, image clarity, resolution, labeling consistency, and class balance, based on data completeness, image quality score, visual inspection, and model performance. DenseNet201 was trained and validated on each dataset achieving its highest accuracy with the Mbozi dataset at 98.72% and a validation accuracy of 97.65%, demonstrating the importance of consistent image quality and accurate annotations. In contrast, the public dataset suffered from inconsistencies in resolution and labeling, resulting in a lower training and validation accuracy of 96.86% and 96.42% respectively. The Combined dataset, which integrated the strengths of both datasets, exhibited a stronger generalization with an accuracy of 97.48% and validation accuracy of 97.49%, balancing the need for high-quality images with environmental variability. The study shows improved CLR detection speed and accuracy due to high-quality and consistently labeled images from the Mbozi dataset. It recommends future models integrate regionally relevant and high-resolution datasets for robust performance in real-world agricultural conditions, providing coffee farmers with timely disease intervention tools for better production management and economic stability in coffee-growing regions.

Balanced hydropower and ecological benefits in reservoir-river-lake system: An integrated framework with machine learning and game theory.

The negative impacts of large hydroelectric reservoirs on downstream ecosystems have attracted worldwide attention. Few attempts have been made to dynamically predict ecological benefits and rationally negotiation in the reservoir-river-lake (RRL) system. This study addresses these gaps by developing an integrated framework with machine learning and game theory to balanced hydropower and ecological benefits. The proposed framework integrated the RRL system simulation with a bargaining model, utilizing a machine learning model to forecast lake levels and the equivalent factor method to assess downstream ecosystem service values (ESV). The study evaluated the framework's generalizability and accuracy by applying random and actual runoff series within the Three Gorges Reservoir to the Dongting Lake region. The 2022 mega-drought case study revealed that the Nash equilibrium operation could simultaneously enhance hydropower generation (7.55%) and ecological benefits (20.00%). Notably, ESV improvements of 61.58% during the post-flood season and 36.07% during the dry season underscored the framework's effectiveness in elevating the ecological benefits. The comparison with traditional multi-objective optimization showed that the proposed framework provided reliable and acceptable solutions for decision-makers. The dynamic weight change elucidates the intricate interactions between economic and ecological benefits, enabling a nuanced adjustment of the single weights in the traditional framework. In addition to enhancing the theoretical framework, the Nash equilibrium solutions also showed positive effects on the carbon cycle, plant growth, and animal habitats in Dongting Lake, further highlighting the practical significance. This research offers a practical management tool for reconciling the conflict in an RRL integrated system, providing theoretical and practical insights into sustainable environment and ecosystem management.

Autonomous learning of generative models with chemical reaction network ensembles.

Can a micron-sized sack of interacting molecules autonomously learn an internal model of a complex and fluctuating environment? We draw insights from control theory, machine learning theory, chemical reaction network theory and statistical physics to develop a general architecture whereby a broad class of chemical systems can autonomously learn complex distributions. Our construction takes the form of a chemical implementation of machine learning's optimization workhorse: gradient descent on the relative entropy cost function, which we demonstrate can be viewed as a form of integral feedback control. We show how this method can be applied to optimize any detailed balanced chemical reaction network and that the construction is capable of using hidden units to learn complex distributions.

Predicting inflammatory response of biomimetic nanofibre scaffolds for tissue regeneration using machine learning and graph theory.

Tissue regeneration after a wound occurs through three main overlapping and interrelated stages namely inflammatory, proliferative, and remodelling phases, respectively. The inflammatory phase is key for successful tissue reconstruction and triggers the proliferative phase. The macrophages in the non-healing wounds remain in the inflammatory loop, but their phenotypes can be changed via interactions with nanofibre-based scaffolds mimicking the organisation of the native structural support of healthy tissues. However, the organisation of extracellular matrix (ECM) is highly complex, combining order and disorder, which makes it difficult to replicate. The possibility of predicting the desirable biomimetic geometry and chemistry of these nanofibre scaffolds would streamline the scaffold design process. Fifteen families of nanofibre scaffolds, electrospun from combinations of polyesters (polylactide, polyhydroxybutyrate), polysaccharides (polysucrose, carrageenan, cellulose), and polyester ether (polydioxanone) were investigated and analysed using machine learning (ML). The Random Forest model had the best performance (92.8%) in predicting inflammatory responses of macrophages on the nanoscaffolds using tumour necrosis factor-alpha as the output. CellProfiler proved to be an effective tool to process scanning electron microscopy (SEM) images of the macrophages on the scaffolds, successfully extracting various features and measurements related to cell phenotypes M0, M1, and M2. Deep learning modelling indicated that convolutional neural network models have the potential to be applied to SEM images to classify macrophage cells according to their phenotypes. The complex organisation of the nanofibre scaffolds can be analysed using graph theory (GT), revealing the underlying connectivity patterns of the nanofibres. Analysis of GT descriptors showed that the electrospun membranes closely mimic the connectivity patterns of the ECM. We conclude that ML-facilitated, GT-quantified engineering of cellular scaffolds has the potential to predict cell interactions, streamlining the pipeline for tissue engineering.

From data to discovery: Exploring novel photovoltaic perovskite chalcogenide crystals MgMS3 (M: Zr, Ti, Hf) via machine learning and density functional theory calculations

From data to discovery: Exploring novel photovoltaic perovskite chalcogenide crystals MgMS3 (M: Zr, Ti, Hf) via machine learning and density functional theory calculations

Intelligent video recommendation and interactive e-learning mode for English teaching courses based on machine learning

With the continuous deepening of intelligent education in China, English education is receiving increasing attention, and educational reform is urgent. However, English education in China is still in the early stages of intelligence, and the current educational resources lack uniformity, teaching methods are single, and the standards for measuring educational quality also need to be improved. Therefore, this article studies the intelligent recommendation system for English teaching course videos based on machine learning. Firstly, the relevant theories of machine learning were analyzed, and the multiple linear regression, recurrent neural network algorithms, and logistic regression models in machine learning technology were analyzed. At the same time, Markov decision processes are also used to reinforce machine learning algorithms. Secondly, an English teaching video recommendation platform was established to analyze the regional preference and style preference in the field of video recommendation. Finally, an English course recommendation system was developed through the video recommendation technology of machine learning and the current situation of English teaching courses, which can promote the innovation and application of English teaching models.

Unified regularity measures for sample-wise learning and generalization

Fundamental machine learning theory shows that different samples contribute unequally to both the learning and testing processes. Recent studies on deep neural networks (DNNs) suggest that such sample differences are rooted in the distribution of intrinsic pattern information, namely sample regularity. Motivated by recent discoveries in network memorization and generalization, we propose a pair of sample regularity measures with a formulation-consistent representation for both processes. Specifically, the cumulative binary training/generalizing loss (CBTL/CBGL), the cumulative number of correct classifications of the training/test sample within the training phase, is proposed to quantify the stability in the memorization-generalization process, while forgetting/mal-generalizing events (ForEvents/MgEvents), i.e., the misclassification of previously learned or generalized samples, are utilized to represent the uncertainty of sample regularity with respect to optimization dynamics. The effectiveness and robustness of the proposed approaches for mini-batch stochastic gradient descent (SGD) optimization are validated through sample-wise analyses. Further training/test sample selection applications show that the proposed measures, which share the unified computing procedure, could benefit both tasks.

Bifurcation Analysis in Dynamical Systems Through Integration of Machine Learning and Dynamical Systems Theory

Abstract Characterizing the nonlinear behavior of dynamical systems near the stability boundary is a critical step toward understanding, designing, and controlling systems prone to stability concerns. Traditional methods for bifurcation analysis in both experimental systems and large-dimensional models are often hindered either by the absence of an accurate model or by the analytical complexity involved. This paper presents a novel approach that combines the theoretical frameworks of nonlinear reduced-order modeling and stability analysis with advanced machine learning techniques to perform bifurcation analysis in dynamical systems. By focusing on a low-dimensional nonlinear invariant manifold, this work proposes a data-driven methodology that simplifies the process of bifurcation analysis in dynamical systems. The core of our approach lies in utilizing carefully designed neural networks to identify nonlinear transformations that map observation space into reduced manifold coordinates in its normal form where bifurcation analysis can be performed. The unique integration of analytical and data-driven approaches in the proposed method enables the learning of these transformations and the performance of bifurcation analysis with a limited number of trajectories. Therefore, this approach improves bifurcation analysis in model-less experimental systems and cost-sensitive high-fidelity simulations. The effectiveness of this approach is demonstrated across several examples.

Parametric Properties of New F-Divergence Measure

Abstract F-divergences are a class of functions that quantify the difference between two probability distributions. They are widely used in statistics, machine learning, and information theory. New F-divergences typically refer to variations or extensions of existing F-divergences, introducing additional parameters to customize their behavior or properties. The parametric properties of a new F-divergence measure would depend on the specific form of the divergence and the parameters introduced. In this paper, discussed the parametric properties of new f-divergence functional. This measure is also known as Jain and Saraswat measure which is established in 2012 and 2013. The applications of this measure in the form of series using the properties of convexity have been established.

Prediction of Rock Fracture Pressure in Hydraulic Fracturing with Interpretable Machine Learning and Mechanical Specific Energy Theory

Prediction of Rock Fracture Pressure in Hydraulic Fracturing with Interpretable Machine Learning and Mechanical Specific Energy Theory

Synergies between Machine Learning, Artificial Intelligence, and Game Theory for Complex Decision-Making

The special focus of this paper is to discuss a likely intersection of machine learning, artificial intelligence (AI) technology, and game theory, pointing at the importance of this synthesis both in mathematics and engineering. As these domains develop various means of addressing decision-making problems become more and more sophisticated and can be used in different areas such as economics, security, and social sciences. We will also address selected game-theoretic issues including the concept of decision making in terms of Nash equilibria or in the distinction of games as being cooperative or non-cooperative and how they work in synergy with machine learning approaches bringing in reinforcements and deep learning to leverage forecasting and strategizing. The paper makes references to problem-oriented branches of studies such as autonomous systems or market strategies stressing the importance of the novel direction for further studies. Within the scope of machine learning and game theory the goal is to implement better complex models which utilize real world intricacies for enhancing decision making within a populated agent environment.

Concrete Creep Prediction Based on Improved Machine Learning and Game Theory: Modeling and Analysis Methods

Understanding the impact of creep on the long-term mechanical features of concrete is crucial, and constructing an accurate prediction model is the key to exploring the development of concrete creep under long-term loads. Therefore, in this study, three machine learning (ML) models, a Support Vector Machine (SVM), Random Forest (RF), and Extreme Gradient Boosting Machine (XGBoost), are constructed, and the Hybrid Snake Optimization Algorithm (HSOA) is proposed, which can reduce the risk of the ML model falling into the local optimum while improving its prediction performance. Simultaneously, the contributions of the input features are ranked, and the optimal model’s prediction outcomes are explained through SHapley Additive exPlanations (SHAP). The research results show that the optimized SVM, RF, and XGBoost models increase their accuracies on the test set by 9.927%, 9.58%, and 14.1%, respectively, and the XGBoost has the highest precision in forecasting the concrete creep. The verification results of four scenarios confirm that the optimized model can precisely capture the compliance changes in long-term creep, meeting the requirements for forecasting the nature of concrete creep.

Equilibria of large random Lotka-Volterra systems with vanishing species: a mathematical approach.

Ecosystems with a large number of species are often modelled as Lotka-Volterra dynamical systems built around a large interaction matrix with random part. Under some known conditions, a global equilibrium exists and is unique. In this article, we rigorously study its statistical properties in the large dimensional regime. Such an equilibrium vector is known to be the solution of a so-called Linear Complementarity Problem. We describe its statistical properties by designing an Approximate Message Passing (AMP) algorithm, a technique that has recently aroused an intense research effort in the fields of statistical physics, machine learning, or communication theory. Interaction matrices based on the Gaussian Orthogonal Ensemble, or following a Wishart distribution are considered. Beyond these models, the AMP approach developed in this article has the potential to describe the statistical properties of equilibria associated to more involved interaction matrix models.

Uncovering migration systems through spatio-temporal tensor co-clustering

A central problem in the study of human mobility is that of migration systems. Typically, migration systems are defined as a set of relatively stable movements of people between two or more locations over time. While these emergent systems are expected to vary over time, they ideally contain a stable underlying structure that could be discovered empirically. There have been some notable attempts to formally or informally define migration systems. However, they have been limited by being hard to operationalize and defining migration systems in ways that ignore origin/destination aspects and fail to account for migration dynamics over time. In this work, we propose to employ spatio-temporal tensor co-clustering—that stems from signal processing and machine learning theory—as a novel migration system analysis tool. Tensor co-clustering is designed to cluster entities exhibiting similar patterns across multiple modalities and thus suits our purpose of analyzing spatial migration activities across time. To demonstrate its effectiveness in describing stable migration systems, we first focus on domestic migration between counties in the US from 1990 to 2018. We conduct three case studies on domestic migration, namely, (i) US Metropolitan Areas, (ii) the state of California, and (iii) Louisiana, in which the last focuses on detecting exogenous events such as Hurricane Katrina in 2005. In addition, we also examine a case study at a larger scale, using worldwide international migration data from 200 countries between 1990 and 2015. Finally, we conclude with a discussion of this approach and its limitations.

Analysis of the Application of Artificial Intelligence in Mahjong

Over the past few years, AI-based models for Mahjong have received a substantial amount of contribution due to the advancements in machine learning and game theory. In this review, these state-of-the-art AI models are considered, including Tjong, Kanachan, Suphx, and Zhejiang University’s developed model. The deployed approaches are disparately represented by reinforcement learning, Monte Carlo tree search (MCTS), and deep neural networks, all of which aim at solving the issues that come with the game’s structure, such as incomplete and overburdening information. The Tjong model complements the power of the transformer architecture with hierarchical decision-making and in turn, brings strategic depth to the gameplay. Kanachan employs Q-learning and MCTS to optimize decision-making, while Suphx combines these methods with the novel ideas of Double Q-learning and Thompson Sampling to achieve higher performance than seen before. The integration of reinforcement learning and a new evaluation model imbues the model with the unique properties of both learning machines and human expertise, namely depth and intelligence. As a result of all these exploits, obstacles still exist, the most notable among them are the management of completely unknown information, handling long-distance games, and the strategic balance between offense and defense. In the future, determinants of probabilistic reasoning, the improvement of deep learning systems, and the prowess of reward shaping will result in AI improvement in Mahjong.

Application of Machine Learning in Human-Computer Intelligent Confrontation

Human-computer intelligent confrontation is a widely used technique in a lot of game scenarios, and the artificial intelligence makes a lot of success in different fields. The level of artificial intelligence improves quickly with the development of machine learning. Machine learning refers to enhancing the capabilities of artificial intelligence through the interaction of machines and data. Among all kinds of machine learning, the most famous one is deep reinforced learning. This technology is currently widely used in human-computer confrontations. This article introduces the technical principles of machine learning and reviews the development history of machine learning theory. Besides, this article introduces the application of machine learning in the two important competitive fields of Go and Texas Hold’em, as well as its development and achievements in these two areas. This article can enable other scholars to have a more comprehensive and systematic understanding of machine learning and its applications in Go and Texas Hold’em.

Leveraging machine learning to expedite screening of single-atom catalysts in electrochemical nitrate reduction to ammonia

The electrochemical reduction of nitrate to ammonia, which is a potentially valuable process for pollution mitigation and NH3 production, faces challenges in developing high-performance catalysts. By integrating machine learning (ML) and density functional theory (DFT), we successfully established ML models using thousands of calculated properties (formation energy, NO3– adsorption, and Gibbs free energy of the critical step) from the literature and predicted the catalytic performance of 1891 previously unknown single-atom catalysts (SACs) for the nitrate reduction reaction (NO3RR), employing a four-step screening strategy (stability, NO3– adsorption, activity, and selectivity) and identifying 10 promising materials. We then verified the properties of these SACs, which exceeded the benchmark materials, thereby validating the ML model, and elucidated the origin of their high NO3RR activity based on their inner electronic structures. This study not only presents prominent candidates and novel descriptors for NO3RR but also establishes an efficient, rapid, and cost-effective methodology for the discovery and design of more valuable SACs.

Boosting tree with bootstrap technique for pre-stress design in cable dome structures

Tensegrity structures, known for their rigidity derived from feasible pre-stresses, present unique challenges in structural engineering. Traditional force-finding methods, though comprehensive, rely heavily on intricate matrix computations, making them computationally intensive and often uncomfortable for considering external loads in practical engineering scenarios. This paper introduces a novel approach to compute pre-stresses in cable dome structures by integrating machine learning and probability theory, collectively termed the boosting tree with bootstrap technique (BTWBT). This method reduces the sample size to as few as 100 per iteration, while improving computational efficiency by randomly generating internal forces. By reframing the force determination as an inverse problem, it ensures that structural displacement converges to zero under feasible pre-stresses. The effectiveness of BTWBT is demonstrated across three distinct cable dome structures: the Geiger dome, Kiewitt dome, and rotating hyperboloid cable dome. Results show that BTWBT achieves the preset displacement requirement (maximum nodal displacement below 0.01 mm) with fewer iterations and reduced computational cost compared to traditional machine learning methods. BTWBT's capability to manage complex structural configurations with minimal data, while incorporating random internal force generation ranges, highlights its potential as a superior alternative for force determination in tensegrity structures.

Underwater Small Target Classification Using Sparse Multi-View Discriminant Analysis and the Invariant Scattering Transform

Sonar automatic target recognition (ATR) systems suffer from complex acoustic scattering, background clutter, and waveguide effects that are ever-present in the ocean. Traditional signal processing techniques often struggle to distinguish targets when noise and complicated target geometries are introduced. Recent advancements in machine learning and wavelet theory offer promising directions for extracting informative features from sonar return data. This work introduces a feature extraction and dimensionality reduction technique using the invariant scattering transform and Sparse Multi-view Discriminant Analysis for identifying highly informative features in the PONDEX09/PONDEX10 datasets. The extracted features are used to train a support vector machine classifier that achieves an average classification accuracy of 97.3% using six unique targets.