Practical Machine Learning Research Articles

Hyperparameter selection for dataset-constrained semantic segmentation: Practical machine learning optimization.

This paper provides a pedagogical example for systematic machine learning optimization in small dataset image segmentation, emphasizing hyperparameter selections. A simple process is presented for medical physicists to examine hyperparameter optimization. This is also applied to a case-study, demonstrating the benefit of the method. An unrestricted public Computed Tomography (CT) dataset, with binary organ segmentation, was used to develop a multiclass segmentation model. To start the optimization process, a preliminary manual search of hyperparameters was conducted and from there a grid search identified the most influential result metrics. A total of 658 different models were trained in 2100h, using 13160 effective patients. The quantity of results was analyzed using random forest regression, identifying relative hyperparameter impact. Metric implied segmentation quality (accuracy 96.8%, precision 95.1%) and visual inspection were found to be mismatched. In this work batch normalization was most important, but performance varied with hyperparameters and metrics selected. Targeted grid-search optimization and random forest analysis of relative hyperparameter importance, was an easily implementable sensitivity analysis approach. The proposed optimization method gives a systematic and quantitative approach to something intuitively understood, that hyperparameters change model performance. Even just grid search optimization with random forest analysis presented here can be informative within hardware and data quality/availability limitations, adding confidence to model validity and minimize decision-making risks. By providing a guided methodology, this work helps medical physicists to improve their model optimization, irrespective of specific challenges posed by datasets and model design.

Data-driven robust optimization in the face of large-scale datasets: An incremental learning approach

One of the most significant current discussions in optimization under deep uncertainty is integrating machine learning and data science into robust optimization, which has led to the emergence of a new field called Data-Driven Robust Optimization (DDRO). When creating data-driven uncertainty sets, it considers a dataset’s complexity, hidden information, and inherent form. One of the more practical machine learning algorithms for creating data-driven uncertainty sets is support vector clustering (SVC). This algorithm has no prerequisites for preliminary information to generate uncertainty sets with arbitrary geometry. More scenarios can reduce risk when developing SVC-based uncertainty sets. However, the lack of a systematic way to manage the large number of these scenarios hinders the employment of SVC. This paper puts forward an incremental learning algorithm based on support vector clustering, called Incremental Support Vector Clustering (ISVC), to construct an uncertainty set incrementally and efficiently using large datasets. This approach’s novelty and main contributions include incrementally constructing uncertainty sets and dynamic management of outliers. In order to update the temporarily stored Bounded Support Vectors (BSV) and identify outliers, the idea of BSV-archive is offered, where the revision-and-recycle operation is tailored to do just that. As a result, some of the newly acquired information is preserved. Experiments on large-scale datasets demonstrate that the proposed ISVC approach can create an uncertainty set comparable to that of an SVC-based method while using significantly less time.

Optimal operation strategy predictive control for an integrated radiant cooling with fresh air system based on machine learning

Radiant cooling systems are widely valued for their great comfort and energy-saving potential. However, they still face the risk of condensation in the early stages of operation, especially in case of random occupancy and intermittent operation. This study aims to avoid unacceptable discomfort durations in randomly occupied rooms that installed integrated radiant cooling and fresh air system while consuming as little energy as possible. This paper firstly compares the effects of adopting only the setback or standby cooling strategy in a randomly occupied conference room by simulation. The simulation results demonstrate the necessity for predicting optimal operation strategy. Subsequently, optimal operation strategy predictive models were built using three machine learning algorithms on three datasets. The evaluation results of the models indicate the feasibility of using data from neighbouring cities to improve the generalisation ability of the target city model. Finally, the best one of models was used to predict optimal operation strategy and achieved good results: discomfort durations of 97.56% of the conferences were within the acceptable range. Additionally, compared to only adopting the standby cooling strategy, the radiant cooling system operating time was reduced by 8.88%, and the total energy consumption was reduced by 28.85 kWh. In this study, a model to predict optimal operation strategy is proposed. By simulating and analysing from both comfort and energy perspectives, the optimal operation strategy predictive control method effectively limits the discomfort duration and reduces energy consumption and radiant system runtime. This study provides an example of practical engineering and machine learning applications for radiant cooling systems.

A practical machine learning approach for predicting the quality of 3D (bio)printed scaffolds

3D (Bio)printing is a highly effective method for fabricating tissue engineering scaffolds, renowned for their exceptional precision and control. Artificial intelligence (AI) has become a crucial technology in this field, capable of learning and replicating complex patterns that surpass human capabilities. However, the integration of AI in tissue engineering is often hampered by the lack of comprehensive and reliable data. This study addresses these challenges by providing one of the most extensive datasets on 3D-printed scaffolds. It provides the most comprehensive open-source dataset and employs various AI techniques, from unsupervised to supervised learning. This dataset includes detailed information on 1171 scaffolds, featuring a variety of biomaterials and concentrations-including 60 biomaterials such as natural and synthesized biomaterials, crosslinkers, enzymes, etc.-along with 49 cell lines, cell densities, and different printing conditions. We used over 40 machine learning and deep learning algorithms, tuning their hyperparameters to reveal hidden patterns and predict cell response, printability, and scaffold quality. The clustering analysis using KMeans identified five distinct ones. In classification tasks, algorithms such as XGBoost, Gradient Boosting, Extra Trees Classifier, Random Forest Classifier, and LightGBM demonstrated superior performance, achieving higher accuracy and F1 scores. A fully connected neural network with six hidden layers from scratch was developed, precisely tuning its hyperparameters for accurate predictions. The developed dataset and the associated code are publicly available onhttps://github.com/saeedrafieyan/MLATEto promote future research.

Practical machine learning with PyTorch

Practical machine learning with PyTorch

Machine Learning Applications of Quantum Computing: A Review

At the intersection of quantum computing and machine learning, this review paper explores the transformative impact these technologies are having on the capabilities of data processing and analysis, far surpassing the bounds of traditional computational methods. Drawing upon an in-depth analysis of 32 seminal papers, this review delves into the interplay between quantum computing and machine learning, focusing on transcending the limitations of classical computing in advanced data processing and applications. This review emphasizes the potential of quantum-enhanced methods in enhancing cybersecurity, a critical sector that stands to benefit significantly from these advancements. The literature review, primarily leveraging Science Direct as an academic database, delves into the transformative effects of quantum technologies on machine learning, drawing insights from a diverse collection of studies and scholarly articles. While the focus is primarily on the growing significance of quantum computing in cybersecurity, the review also acknowledges the promising implications for other sectors as the field matures. Our systematic approach categorizes sources based on quantum machine learning algorithms, applications, challenges, and potential future developments, uncovering that quantum computing is increasingly being implemented in practical machine learning scenarios. The review highlights advancements in quantum-enhanced machine learning algorithms and their potential applications in sectors such as cybersecurity, emphasizing the need for industry-specific solutions while considering ethical and security concerns. By presenting an overview of the current state and projecting future directions, the paper sets a foundation for ongoing research and strategic advancement in quantum machine learning.

Multi-output neural network model for predicting biochar yield and composition

In biomass pyrolysis for biochar production, existing prediction models face computational challenges and limited accuracy. This study curated a comprehensive dataset, revealing pyrolysis parameters' dominance in biochar yield (54.8 % importance). Pyrolysis temperature emerged as pivotal (PCC = −0.75), influencing yield significantly. Artificial Neural Network (ANN) outperformed Random Forest (RF) in testing set predictions (R2 = 0.95, RMSE = 3.6), making it apt for complex multi-output predictions and software development. The trained ANN model, employed in Partial Dependence Analysis, uncovered nonlinear relationships between biomass characteristics and biochar yield. Findings indicated optimization opportunities, correlating low pyrolysis temperatures, elevated nitrogen content, high fixed carbon, and brief residence times with increased biochar yields. A multi-output ANN model demonstrated optimal fit for biochar yield. A user-friendly Graphical User Interface (GUI) for biochar synthesis prediction was developed, exhibiting robust performance with a mere 0.52 % prediction error for biochar yield. This study showcases practical machine learning application in biochar synthesis, offering valuable insights and predictive tools for optimizing biochar production processes.

Machine Learning the Disorder Landscape of Majorana Nanowires.

We develop a practical machine learning approach to determine the disorder landscape of Majorana nanowires by using training of the conductance matrix and inverting the conductance data in order to obtain the disorder details in the system. The inversion carried out through machine learning using different disorder parametrizations turns out to be unique in the sense that any input tunnel conductance as a function of chemical potential and Zeeman energy can indeed be inverted to provide the correct disorder landscape. Our work opens up a qualitatively new direction of directly determining the topological invariant and the Majorana wave-function structure corresponding to a transport profile of a device using simulations that quantitatively match the specific conductance profile. In addition, this also opens up the possibility for optimizing Majorana systems by figuring out the (generally unknown) underlying disorder only through the conductance data. An accurate estimate of the applicable spin-orbit coupling in the system can also be obtained within the same scheme.

STUCK PIPE MITIGATION: FINDINGS, ANALYSIS, AND MACHINE LEARNING SOLUTIONS

The occurrence of stuck pipes during drilling presents significant challenges, leading to nonproductive time (NPT) and potential damage to equipment. This paper delves into the reasons behind, effects of, and methods for addressing incidents of stuck pipes in drilling actions. Extensive literature review and analysis reveal the prevalence of stuck pipe events and their diverse underlying causes, ranging from differential sticking to mechanical jamming. Additionally, the study identifies key drilling parameters and signals indicative of potential stuck pipe incidents, emphasizing the importance of early detection and intervention to prevent adverse outcomes. Proposing the utilization of machine learning algorithms and statistical models for predictive analytics as a proactive strategy to mitigate the risk of stuck pipe incidents. The paper discusses the limitations of traditional statistical models and highlights the efficacy of machine learning methods, such as artificial neural networks (ANNs) and support vector machines (SVMs), in accurately modeling drilling processes. Furthermore, practical machine learning models using real-time drilling data are presented as effective tools for early detection and timely intervention in stuck pipe incidents. The study also describes various software tools used to assess and mitigate risks in drilling operations. Overall, this paper presents prospects for improving operational efficiency and safety of drilling operations using advanced analytical methods and software solutions. Keywords: stuck pipe, type of stuck, drillstring immobilization, nonproductive time, machine learning in drilling, risk analysis software.

Instance-Conditional Timescales of Decay for Non-Stationary Learning

Slow concept drift is a ubiquitous, yet under-studied problem in practical machine learning systems. In such settings, although recent data is more indicative of future data, naively prioritizing recent instances runs the risk of losing valuable information from the past. We propose an optimization-driven approach towards balancing instance importance over large training windows. First, we model instance relevance using a mixture of multiple timescales of decay, allowing us to capture rich temporal trends. Second, we learn an auxiliary scorer model that recovers the appropriate mixture of timescales as a function of the instance itself. Finally, we propose a nested optimization objective for learning the scorer, by which it maximizes forward transfer for the learned model. Experiments on a large real-world dataset of 39M photos over a 9 year period show upto 15% relative gains in accuracy compared to other robust learning baselines. We replicate our gains on two collections of real-world datasets for non-stationary learning, and extend our work to continual learning settings where, too, we beat SOTA methods by large margins.

A machine learning classifier for automated fault detection and diagnosis (AFDD) of rooftop units, addressing practical challenges of application

There have been many recent developments to take advantage of machine learning for the task of automated fault detection and diagnosis in air-conditioning equipment. A common approach is to develop a machine learning model to classify fault status, using simulated or measurement data, and test the model’s fault diagnostic capability on the same system that was used for training. However, such models don’t generalize well to different systems. Compounding this challenge, machine learning requires a rich dataset to train the models, and many of the measurement features require additional sensors, all making the potential for practical machine learning AFDD questionable. The current paper focuses on these challenges by: (i) training a model using simulation data from three rooftop units (RTU); (ii) rebalancing the training dataset to reduce false alarms; (iii) developing a feature set that minimizes sensor cost without significantly impacting performance; (iv) testing the model’s performance on a set of laboratory measurement data from the same RTUs that were simulated for training data; (v) testing the model on a faulted RTU in the field. The results demonstrate good progress toward practical application of machine learning based AFDD for RTU, but generalization to different systems remains a challenge.

Distributed quantum neural networks via partitioned features encoding

Quantum neural networks are expected to be a promising application in near-term quantum computing, but face challenges such as vanishing gradients during optimization and limited expressibility by a limited number of qubits and shallow circuits. To mitigate these challenges, an approach using distributed quantum neural networks has been proposed to make a prediction by approximating outputs of a large circuit using multiple small circuits. However, the approximation of a large circuit requires an exponential number of small circuit evaluations. Here, we instead propose to distribute partitioned features over multiple small quantum neural networks and use the ensemble of their expectation values to generate predictions. To verify our distributed approach, we demonstrate ten class classification of the Semeion and MNIST handwritten digit datasets. The results of the Semeion dataset imply that while our distributed approach may outperform a single quantum neural network in classification performance, excessive partitioning reduces performance. Nevertheless, for the MNIST dataset, we succeeded in ten class classification with exceeding 96% accuracy. Our proposed method not only achieved highly accurate predictions for a large dataset but also reduced the hardware requirements for each quantum neural network compared to a large single quantum neural network. Our results highlight distributed quantum neural networks as a promising direction for practical quantum machine learning algorithms compatible with near-term quantum devices. We hope that our approach is useful for exploring quantum machine learning applications.

Stochastic gradient descent with random label noises: doubly stochastic models and inference stabilizer

Random label noise (or observational noise) widely exists in practical machine learning settings. While previous studies primarily focused on the effects of label noise to the performance of learning, our work intends to investigate the implicit regularization effects of label noise, under mini-batch sampling settings of stochastic gradient descent (SGD), with the assumption that label noise is unbiased. Specifically, we analyze the learning dynamics of SGD over the quadratic loss with unbiased label noise (ULN), where we model the dynamics of SGD as a stochastic differentiable equation with two diffusion terms (namely a doubly stochastic model). While the first diffusion term is caused by mini-batch sampling over the (label-noiseless) loss gradients, as in many other works on SGD (Zhu et al 2019 ICML 7654–63; Wu et al 2020 Int. Conf. on Machine Learning (PMLR) pp 10367–76), our model investigates the second noise term of SGD dynamics, which is caused by mini-batch sampling over the label noise, as an implicit regularizer. Our theoretical analysis finds such an implicit regularizer would favor some convergence points that could stabilize model outputs against perturbations of parameters (namely inference stability). Though similar phenomenon have been investigated by Blanc et al (2020 Conf. on Learning Theory (PMLR) pp 483–513), our work does not assume SGD as an Ornstein–Uhlenbeck-like process and achieves a more generalizable result with convergence of the approximation proved. To validate our analysis, we design two sets of empirical studies to analyze the implicit regularizer of SGD with unbiased random label noise for deep neural network training and linear regression. Our first experiment studies the noisy self-distillation tricks for deep learning, where student networks are trained using the outputs from well-trained teachers with additive unbiased random label noise. Our experiment shows that the implicit regularizer caused by the label noise tends to select models with improved inference stability. We also carry out experiments on SGD-based linear regression with ULN, where we plot the trajectories of parameters learned in every step and visualize the effects of implicit regularization. The results back up our theoretical findings.

Comparative Analysis of Machine Learning Models for Accurate Retail Sales Demand Forecasting

This article compares sales forecasting models, LSTM and LGBM, using retail sales data from an American multinational company. The study employs a meticulous methodology, optimizing memory, performing feature engineering, and adjusting model parameters for both LSTM and LGBM. Evaluation metrics, including RMSE, MAE, WMAPE, and WRMSEE, demonstrate that LGBM consistently outperforms LSTM in capturing and predicting sales patterns. The analysis favors LGBM as the preferred model for retail sales demand forecasting, emphasizing the importance of model selection. This study contributes to practical machine learning applications in retail sales forecasting, highlighting LGBM as an effective choice.

A TinyML Model for Gesture-Based Air Handwriting Arabic Numbers Recognition

In an era where the demand for efficient and practical machine learning (ML) solutions on resource-constrained devices is evergrowing, the realm of tiny machine learning (TinyML) emerges as a promising frontier. Motivated by the need for lightweight, low-power models that can be deployed on edge devices, this research paper presents an innovative TinyML model tailored to recognize Arabic hand gestures executed in mid-air. With a primary emphasis on the precise classification of Arabic numbers through these expressive hand movements, the paper unveils a comprehensive dataflow architecture. This intricate architecture processes accelerometer and gyroscope data to derive exact 2D gesture coordinates, a fundamental component of the recognition process. The cornerstone of the proposed model is the integration of Convolutional Neural Networks (CNNs), elucidating their exceptional role in achieving an impressive 93.8% accuracy rate in the classification of diverse Arabic Numbers gestures. This remarkable level of precision underscores the model's efficacy and resilience, rendering it an ideal candidate for real-time deployment in various gesture recognition scenarios.

Data-driven robust optimization based on position-regulated support vector clustering

The changes experienced in integrating machine learning and data science into mathematical programming over the past decade remain unprecedented. They have created a novel optimization component under deep uncertainty known as “Data-Driven Robust Optimization” (DDRO). It considers a dataset's complexity, hidden information, and inherent form when creating data-driven uncertainty sets. One of the more practical machine learning algorithms for creating data-driven uncertainty sets is support vector clustering (SVC). There are no prerequisites for preliminary information to generate uncertainty sets with arbitrary geometry. Moreover, it can regularize the conservatism level and adjust a balance between the magnitude of the uncertainty set and information loss by a control parameter. Despite being superior and effective, SVC needs help selecting the best value for the trade-off parameter. This parameter impacts the robustness of the solution and the ability of SVC to handle information loss; hence, it must be accurately determined. Irrational data elimination as outliers alters the degree of conservatism, resulting in information loss and suboptimal solutions. This paper seeks to remedy this problem by suggesting position-regulated support vector clustering (PRSVC) to involve the importance of each data point in estimating the trade-off parameter. Experiments on artificial and real-world data collections indicate that, in comparison to the current SVC-based model, the proposed approach offers a more accurate decision boundary.

Development of a machine learning model for on-site evaluation of concrete compressive strength by SonReb

Coring or destructive testing is typically the default choice for the evaluation of concrete compressive strength in reinforced concrete (RC) structures. However, it can be impractical and/or not representative of all areas of interest in a structure. While various non-destructive test (NDT) methods can be correlated to concrete strength, the accuracy of any single NDT used for this purpose is generally low. The SonReb method, which combines ultrasonic pulse velocity readings and rebound number, has been shown to have improved accuracy over single test methods. However, empirical SonReb equations, calibrated to specific datasets using regression analysis, cannot necessarily be applied to concrete from other sources without introducing significant errors. This study presents a practical machine learning (ML) model for on-site concrete strength prediction. A large database was created from available literature along with new experimental test data. Three different ML models based on an adaptive neuro-fuzzy inference system (ANFIS) were developed along with a graphical user interface application to facilitate its use in the field. In addition to the ML models, linear and non-linear regression analyses were also conducted and compared with existing equations in the literature. The accuracy of each model was subsequently validated against core samples extracted from a reinforced concrete slab. The results show that the proposed ML model and non-linear regression provided the most reliable predictions of concrete strength of the validation specimen with a mean absolute error of less than 10 % compared with twelve core samples. The findings suggest that ML can be a potential tool to evaluate the in-place compressive strength of RC structures in combination with simple NDT methods.

Real-time water quality prediction in water distribution networks using graph neural networks with sparse monitoring data

Ensuring the safety and reliability of drinking water supply requires accurate prediction of water quality in water distribution networks (WDNs). However, existing hydraulic model-based approaches for system state prediction face challenges in model calibration with limited sensor data and intensive computing requirements, while current machine learning models are lack of capacity to predict the system states at sites that are not monitored or included in model training. To address these gaps, this study proposes a novel gated graph neural network (GGNN) model for real-time water quality prediction in WDNs. The GGNN model integrates hydraulic flow directions and water quality data to represent the topology and system dynamics, and employs a masking operation for training to enhance prediction accuracy. Evaluation results from a real-world WDN demonstrate that the GGNN model is capable to achieve accurate water quality prediction across the entire WDN. Despite being trained with water quality data from a limited number of sensor sites, the model can achieve high predictive accuracies (Mean Absolute Error = 0.07 mg L−1 and Mean Absolute Percentage Error = 10.0 %) across the entire network including those unmonitored sites. Furthermore, water quality-based sensor placement significantly improves predictive accuracy, emphasizing the importance of careful sensor location selection. This research advances water quality prediction in WDNs by offering a practical and effective machine learning solution to address challenges related to limited sensor data and network complexity. This study provides a first step towards developing machine learning models to replace hydraulic models in WDN modelling.

Analysis of Practical Machine Learning Scenarios for Cybersecurity in Industry 4.0

In recent years, the widespread adoption of Artificial Intelligence (AI) has been propelled by global technological advancements. Some organizations view AI as a potential threat to their cybersecurity, while others regard it as a way to advance and attain cyber resilience. The recent progress in AI has resulted in the publication of various frameworks and regulations, including notable examples such as the EU AI Act. While certain nations and governments are prudently assessing the risks associated with widespread AI deployment, threat actors are already exploiting these emerging opportunities. In this paper, we delve into an examination of the utilization of artificial intelligence methods in the realm of cybersecurity within the context of Industry 4.0. Additionally, we provide an in-depth exploration of the obstacles, motivations, and essential requirements that warrant meticulous consideration when deploying these AI approaches to bolster the cybersecurity landscape in Industry 4.0.

Evaluating machine and deep learning techniques in predicting blood sugar levels within the E-health domain

This paper focuses on exploring and comparing different machine learning algorithms in the context of diabetes management. The aim is to understand their characteristics, mathematical foundations, and practical implications specifically for predicting blood glucose levels. The study provides an overview of the algorithms, with a particular emphasis on deep learning techniques such as Long Short-Term Memory Networks. Efficiency is a crucial factor in practical machine learning applications, especially in the context of diabetes management. Therefore, the paper investigates the trade-off between accuracy, resource utilisation, time consumption, and computational power requirements, aiming to identify the optimal balance. By analysing these algorithms, the research uncovers their distinct behaviours and highlights their dissimilarities, even when their analytical underpinnings may appear similar.