Machine Learning Architecture Research Articles

A novel framework for analyzing internet of things datasets for machine learning and deep learning-based intrusion detection systems

To generate a machine learning (ML) and deep learning (DL) architecture with good performance, we need a decent dataset for the training and testing phases of the development process. Starting with the knowledge discovery and data mining (KDD) Cup 99 dataset, numerous datasets have been produced since 1998 to be utilized in the ML and DL-based intrusion detection systems (IDS) training and testing process. Because there are so many datasets accessible, it might be challenging for researchers to choose which dataset to employ. Therefore, a framework for evaluating dataset appropriateness with the research to be conducted is becoming increasingly crucial as new datasets are regularly created. Additionally, given the growing popularity of internet of things (IoT) devices and an increasing number of specific datasets for IoT in recent years, it is essential to have a specific framework for IoT datasets. Therefore, this research aims to develop a new framework for evaluating IoT datasets for ML and DL-based IDS. The study's findings include, first, a novel framework for assessing IoT datasets, second, a comparison of this novel framework to other existing frameworks, and third, an analysis of five IoT datasets by using the new framework.

Machine Learning and Deep Learning Approach for Medical Image Analysis Summary Generator

Colorectal cancer, which is frequent, recognized tumours in both genders around the globe. As per the report generated by WHO in 2018, colon cancer placed in the third position, whereas 1.80 million individuals are affected. Precisely, it is the succeeding leading cancer, which is the second most common cause of cancer in females, and the third for males. The loss of control over the integrity of epidermal cells in bowel or malignancy can be the cause of colorectal cancer. An effective way to recognize colon cancer at an early stage and substantial treatment can reduce the ensuing death rates to a great extent. To perform Screening of Morphology of Malignant Tumor Cells in the colon, a Gastroenterologist may refer to cancer diagnosis tests for pathological images. In any Histology method, the process takes a significant duration of time due to infinite numbers of glands in the gastrointestinal system, which may lead to irreconcilable outcomes. By diagnosing through computer algorithms, can give practical and contributory results. Hence, accurate gland segmentation is one crucial prerequisite stage to get reliable and informative morphological image data. In recent times, the scholars applied machine learning algorithms to pathological image analysis for the diagnosis of cancer disease. We propose that features extracted from the diagnostic tests, given as input to a machine learning architecture used along with semantic segmentation algorithm, provide results that are accurate than the existing image segmentation algorithms. This work is the extensive review of machine learning architectures used for semantic segmentation on the histological images of the colon.In our project we will be using the following algorithms such as Adaboost as existing and Convolution Neural Network (CNN) as proposed and its accuracy is been calculated and well compared to other algorithms. It is found that CNN performs less than other algorithms

Physics-informed neural networks for high-resolution weather reconstruction from sparse weather stations

Background The accurate provision of weather information holds immense significance to many disciplines. One example corresponds to the field of air traffic management, in which one basis for weather detection is set upon recordings from sparse weather stations on ground. The scarcity of data and their lack of precision poses significant challenges to achieve a detailed description of the atmosphere state at a certain moment in time. Methods In this article, we foster the use of physics-informed neural networks (PINNs), a type of machine learning (ML) architecture which embeds mathematically accurate physics models, to generate high-quality weather information subject to the regularization provided by the Navier-Stokes equations. Results The application of PINNs is oriented to the reconstruction of dense and precise wind and pressure fields in areas where only a few local measurements provided by weather stations are available. Our model does not only disclose and regularize such data, which are potentially corrupted by noise, but is also able to precisely compute wind and pressure in target areas. Conclusions The effect of time and spatial resolution over the capability of the PINN to accurately reconstruct fluid phenomena is thoroughly discussed through a parametric study, concluding that a proper tuning of the neural network’s loss function during training is of utmost importance.

Developing Time Series Forecasting Models with Generative Large Language Models

Nowadays, Generative Large Language Models (GLLMs) have made a significant impact in the field of Artificial Intelligence (AI). One of the domains extensively explored for these models is their ability as generators of functional source code for software projects. Nevertheless, their potential as assistants to write the code needed to generate and model Machine Learning (ML) or Deep Learning (DL) architectures has not been fully explored to date. For this reason, this work focuses on evaluating the extent to which different tools based on GLLMs, such as ChatGPT or Copilot, are able to correctly define the source code necessary to generate viable predictive models. The use case defined is the forecasting of a time series that reports the indoor temperature of a greenhouse. The results indicate that, while it is possible to achieve good accuracy metrics with simple predictive models generated by GLLMs, the composition of predictive models with complex architectures using GLLMs is still far from improving the accuracy of predictive models generated by human data scientists.

U-Plume: automated algorithm for plume detection and source quantification by satellite point-source imagers

Abstract. Current methods for detecting atmospheric plumes and inferring point-source rates from high-resolution satellite imagery are labor-intensive and not scalable with regard to the growing satellite dataset available for methane point sources. Here, we present a two-step algorithm called U-Plume for automated detection and quantification of point sources from satellite imagery. The first step delivers plume detection and delineation (masking) with a U-Net machine learning architecture for image segmentation. The second step quantifies the point-source rate from the masked plume using wind speed information and either a convolutional neural network (CNN) or a physics-based integrated mass enhancement (IME) method. The algorithm can process 62 images (each measuring 128 pixels × 128 pixels) per second on a single 2.6 GHz Intel Core i7-9750H CPU. We train the algorithm using large-eddy simulations of methane plumes superimposed on noisy and variable methane background scenes from the GHGSat-C1 satellite instrument. We introduce the concept of point-source observability, Ops=Q/(UWΔB), as a single dimensionless number to predict plume detectability and source rate quantification error from an instrument as a function of source rate Q, wind speed U, instrument pixel size W, and instrument-dependent background noise ΔB. We show that Ops can powerfully diagnose the ability of an imaging instrument to observe point sources of a certain magnitude under given conditions. U-Plume successfully detects and masks plumes from sources as small as 100 kg h−1 in GHGSat-C1 images over surfaces with low background noise and successfully handles larger point sources over surfaces with substantial background noise. We find that the IME method for source quantification is unbiased over the full range of source rates, while the CNN method is biased towards the mean of its training range. The total error in source rate quantification is dominated by wind speed at low wind speeds and by the masking algorithm at high wind speeds. A wind speed of 2–4 m s−1 is optimal for detection and quantification of point sources from satellite data.

Application of deep learning and inline holography to estimate the droplet size distribution

We examine five machine learning-based architectures to estimate the droplet size distributions obtained using digital inline holography. The architectures, namely, U-Net, R2 U-Net, Attention U-Net, V-Net, and Residual U-Net, are trained using synthetic holographic images. Our assessment focuses on evaluating the training, validation, and prediction performance of these architectures. Our results indicate that U-Net and R2 U-Net are the most proficient architectures, consistently demonstrating superior performance trends and achieving the highest Intersection Over Union (IOU) scores among the five architectures studied. We employ additional training using experimental holographic images for the two top-performing architectures to validate their efficacy further. Subsequently, they are employed to segment an experimental dataset illustrating the bag breakup phenomenon, facilitating the extraction of size distribution. The extracted size distribution from U-Net and R2 U-Net segmentation is then compared with the analytical model proposed by Jackiw and Ashgriz (2022) by employing the gamma and log-normal distributions. Our findings indicate that the gamma distribution provides a more accurate prediction of the multi-modal size distribution than the log-normal distribution owing to its long exponential tail. The present study offers valuable insights into the effectiveness of machine learning architectures in estimating particle/droplet sizes, highlighting their practical application in real-world experimental scenarios.

Mathematics-Informed Neural Network for Acoustic Scattering by Parallel Plates

This paper proposes a mathematics-informed neural network (MINN) approach for resolving the long-term challenge in wave scattering modeling. The central innovation lies in integrating Cauchy–Riemann equations into machine learning architectures. By incorporating Cauchy integrals and boundary conditions, the neural network successfully learns to numerically produce matrix kernel factorization for Wiener–Hopf analytical models. To validate and demonstrate the approach, a benchmark case of wave scattering from parallel hard–soft plates is studied by comparing the machine learning results with the available analytical solutions. The proposed MINN approach could provide a new route to extensively enhance the theoretical modeling capability for several wave scattering and fluid mechanics problems. The code can be found at https://github.com/lscapku/MINN.

ML-SAFT: A machine learning framework for PCP-SAFT parameter prediction

The Perturbed Chain Polar Statistical Associating Fluid Theory (PCP-SAFT) equation of state (EoS) is widely used to predict fluid-phase thermodynamics, but parameterization of PCP-SAFT for individual molecules is often challenging. We propose a machine learning framework called ML-SAFT that can turn experimental data in predictive models of PCP-SAFT parameters. We demonstrate methods for automated large scale regression of PCP-SAFT parameters and thus create a large PCP-SAFT parameter dataset in the literature. We then evaluate several machine learning architectures for predicting PCP-SAFT parameters. We find that our best model provides accurate predictions for a wider range of molecules than existing predictive methods with 40 % average absolute deviation (% AAD) in vapor pressure predictions and 8 % AAD in density predictions.

KbhbXG: A Machine learning architecture based on XGBoost for prediction of lysine β-Hydroxybutyrylation (Kbhb) modification sites

Lysine β-hydroxybutyrylation is an important post-translational modification (PTM) involved in various physiological and biological processes. In this research, we introduce a novel predictor KbhbXG, which utilizes XGBoost to identify β-hydroxybutyrylation modification sites based on protein sequence information. The traditional experimental methods employed for the identification of β-hydroxybutyrylated sites using proteomic techniques are both costly and time-consuming. Thus, the development of computational methods and predictors can play a crucial role in facilitating the rapid identification of β-hydroxybutyrylation sites. Our proposed KbhbXG model first utilizes machine learning algorithm XGBoost to predict β-hydroxybutyrylation modification sites. On the independent test set, KbhbXG achieves an accuracy of 0.7457, specificity of 0.7771, and an impressive area under the curve (AUC) score of 0.8172. The high AUC score achieved by our method demonstrates its potential for effectively identifying novel β-hydroxybutyrylation sites, thereby facilitating further research and exploration of the β-hydroxybutyrylation process. Also, functional analyses have revealed that different organisms preferentially engage in distinct biological processes and pathways, which can provide valuable insights for understanding the mechanism of β-hydroxybutyrylation and guide experimental verification. To promote transparency and reproducibility, we have made both the codes and dataset of KbhbXG publicly available. Researchers interested in utilizing our proposed model can access these resources at https://github.com/Lab-Xu/KbhbXG.

A machine learning architecture for including wave breaking in envelope-type wave models

Wave breaking is a complex physical process about which open questions remain. For some applications, it is critical to include breaking effects in phase-resolved envelope-based wave models such as the non-linear Schrödinger. A promising approach is to use machine learning to capture breaking effects. In the present paper we develop the machine learning architecture to model breaking developed by Eeltink et al. (2022) further, potentially enabling more detailed breaking physics to be captured. We show that this model can be trained on focused wave groups but can also capture breaking in random waves and modulated plane waves. Analysis of the model suggests that the machine learning has broken the problem into two—one part which detects whether the wave is breaking and another which captures the subsequent behaviour, consistent with the way human scientists routinely understand the breaking problem.

A mini-review on perturbation modelling across single-cell omic modalities

Recent advances in single-cell omics technology have transformed the landscape of cellular and molecular research, enriching the scope and intricacy of cellular characterisation. Perturbation modelling seeks to comprehensively grasp the effects of external influences like disease onset or molecular knock-outs or external stimulants on cellular physiology, specifically on transcription factors, signal transducers, biological pathways, and dynamic cell states. Machine and deep learning tools transform complex perturbational phenomena in algorithmically tractable tasks to formulate predictions based on various types of single-cell datasets. However, the recent surge in tools and datasets makes it challenging for experimental biologists and computational scientists to keep track of the recent advances in this rapidly expanding filed of single-cell modelling. Here, we recapitulate the main objectives of perturbation modelling and summarise novel single-cell perturbation technologies based on genetic manipulation like CRISPR or compounds, spanning across omic modalities. We then concisely review a burgeoning group of computational methods extending from classical statistical inference methodologies to various machine and deep learning architectures like shallow models or autoencoders, to biologically informed approaches based on gene regulatory networks, and to combinatorial efforts reminiscent of ensemble learning. We also discuss the rising trend of large foundational models in single-cell perturbation modelling inspired by large language models. Lastly, we critically assess the challenges that underline single-cell perturbation modelling while pointing towards relevant future perspectives like perturbation atlases, multi-omics and spatial datasets, causal machine learning for interpretability, multi-task learning for performance and explainability as well as prospects for solving interoperability and benchmarking pitfalls.

Interpretable machine learning models for failure cause prediction in imbalanced oil pipeline data

Pipelines are critical arteries in the oil and gas industry and require massive capital investment to safely construct networks that transport hydrocarbons across diverse environments. However, these pipeline systems are prone to integrity failure, which results in significant economic losses and environmental damage. Accurate prediction of pipeline failure events using historical oil pipeline accident data enables asset managers to plan sufficient maintenance, rehabilitation, and repair activities to prevent catastrophic failures. However, learning the complex interdependencies between pipeline attributes and rare failure events presents several analytical challenges. This study proposes a novel machine learning (ML) framework to accurately predict pipeline failure causes on highly class-imbalanced data compiled by the United States Pipeline and Hazardous Materials Safety Administration. Natural language processing techniques were leveraged to extract informative features from unstructured text data. Furthermore, class imbalance in the dataset was addressed via oversampling and intrinsic cost-sensitive learning (CSL) strategies adapted for the multi-class case. Nine machine and deep learning architectures were benchmarked, with LightGBM demonstrating superior performance. The integration of CSL yielded an 86% F1 score and a 0.82 Cohen kappa score, significantly advancing prior research. This study leveraged a comprehensive Shapley Additive explanation analysis to interpret the predictions from the LightGBM algorithm, revealing the key factors driving failure probabilities. Leveraging sentiment analysis allowed the models to capture a richer, more multifaceted representation of the textual data. This study developed a novel CSL approach that integrates domain knowledge regarding the varying cost impacts of misclassifying different failure types into ML models. This research demonstrated an effective fusion of text insights from inspection reports with structured pipeline data that enhances model interpretability. The resulting AI modeling framework generated data-driven predictions of the causes of failure that could enable transportation agencies with actionable insights. These insights enable tailored preventative maintenance decisions to proactively mitigate emerging pipeline failures.

Analyzing the Combined Effects of Sarcasm and Emotion for Gender Prediction

“Women are bitchy but men are sarcastic”, such comments reveal the relationship between gender and sarcasm. Automatic gender identification can play a crucial role in services that depend on data about a user’s background. Although for some social media users the gender of a user is typically unavailable due to privacy and anonymity. Based on the notion that male and female users may express their thoughts and sentiments differently in their posts, social media accounts can be examined using their posts (text) in order to automatically identify the gender of an anonymous user. In the current work, efforts are made in analyzing the effects of emotion and sarcasm intended by the users in their tweets for predicting gender. Sarcasm + emotion aided gender prediction systems are developed using different machine learning and neural network-based architectures. In our developed model, tweet features are extracted by using pre-trained GloVe embeddings. The sarcasm intensity is concatenated with the corresponding tweet representation and at last classification layer is used to predict the gender labels. For the experimentation purpose, the PAN-2018 dataset has been used. We have also shown the effect of utilizing emotion, and sarcasm information over gender prediction using different models

Estimation of instantaneous peak flows in Canadian rivers: an evaluation of conventional, nonlinear regression, and machine learning methods.

Instantaneous peak flows (IPFs) are often required to derive design values for sizing various hydraulic structures, such as culverts, bridges, and small dams/levees, in addition to informing several water resources management-related activities. Compared to mean daily flows (MDFs), which represent averaged flows over a period of 24 h, information on IPFs is often missing or unavailable in instrumental records. In this study, conventional methods for estimating IPFs from MDFs are evaluated and new methods based on the nonlinear regression framework and machine learning architectures are proposed and evaluated using streamflow records from all Canadian hydrometric stations with natural and regulated flow regimes. Based on a robust model selection criterion, it was found that multiple methods are suitable for estimating IPFs from MDFs, which precludes the idea of a single universal method. The performance of machine learning-based methods was also found reasonable compared to conventional and regression-based methods. To build on the strengths of individual methods, the fusion modeling concept from the machine learning area was invoked to synthesize outputs of multiple methods. The study findings are expected to be useful to the climate change adaptation community, which currently heavily relies on MDFs simulated by hydrologic models.

Accelerating deep neural network learning using data stream methodology

This paper introduces a novel machine learning approach, called BBATDD (Boosting-Based Algorithm Trained with Drift Detector), that departs from traditional epoch-based methods by adopting a data stream generation strategy from the original dataset. The proposed method is versatile, applicable to a broad range of deep structures and other machine learning architectures. The key to its effectiveness is the conversion of original data into a streaming sequence, which is then fed into the neural network structure. To monitor the learning process and prevent overfitting, a drift detector, commonly used in stream data mining, is employed. Additionally, two methods, ELB and ENT, are introduced to prioritize problematic data elements during the learning process, enhancing overall model performance. Extensive simulations on benchmark datasets (MNIST and CIFAR) validate that BBATDD achieves higher accuracy with significantly fewer batches, without sacrificing generalization capability. For instance, when applied to the MNIST dataset and processing 10 000 batches while comparing loss function values, our approach demonstrates a remarkable 45.6% improvement.

Evaluating the robustness of multimodal task load estimation models

Numerous studies have focused on constructing multimodal machine learning models for estimating a person's cognitive load. However, a prevalent limitation is that these models are typically evaluated on data from the same scenario they were trained on. Little attention has been given to their robustness against data distribution shifts, which may occur during deployment. The aim of this paper is to investigate the performance of these models when confronted with a scenario different from the one on which they were trained. For this evaluation, we utilized a dataset encompassing two distinct scenarios: an n-Back test and a driving simulation. We selected a variety of classic machine learning and deep learning architectures, which were further complemented by various fusion techniques. The models were trained on the data from the n-Back task and tested on both scenarios to evaluate their predictive performance. However, the predictive performance alone may not lead to a trustworthy model. Therefore, we looked at the uncertainty estimates of these models. By leveraging these estimates, we can reduce misclassification by resorting to alternative measures in situations of high uncertainty. The findings indicate that late fusion produces stable classification results across the examined models for both scenarios, enhancing robustness compared to feature-based fusion methods. Although a simple logistic regression tends to provide the best predictive performance for n-Back, this is not always the case if the data distribution is shifted. Finally, the predictive performance of individual modalities differs significantly between the two scenarios. This research provides insights into the capabilities and limitations of multimodal machine learning models in handling distribution shifts and identifies which approaches may potentially be suitable for achieving robust results.

A Comparative Study on Feature Extraction Techniques for the Discrimination of Frontotemporal Dementia and Alzheimer's Disease with Electroencephalography in Resting-State Adults.

Early-stage Alzheimer's disease (AD) and frontotemporal dementia (FTD) share similar symptoms, complicating their diagnosis and the development of specific treatment strategies. Our study evaluated multiple feature extraction techniques for identifying AD and FTD biomarkers from electroencephalographic (EEG) signals. We developed an optimised machine learning architecture that integrates sliding windowing, feature extraction, and supervised learning to distinguish between AD and FTD patients, as well as from healthy controls (HCs). Our model, with a 90% overlap for sliding windowing, SVD entropy for feature extraction, and K-Nearest Neighbors (KNN) for supervised learning, achieved a mean F1-score and accuracy of 93% and 91%, 92.5% and 93%, and 91.5% and 91% for discriminating AD and HC, FTD and HC, and AD and FTD, respectively. The feature importance array, an explainable AI feature, highlighted the brain lobes that contributed to identifying and distinguishing AD and FTD biomarkers. This research introduces a novel framework for detecting and discriminating AD and FTD using EEG signals, addressing the need for accurate early-stage diagnostics. Furthermore, a comparative evaluation of sliding windowing, multiple feature extraction, and machine learning methods on AD/FTD detection and discrimination is documented.

Application of big data analysis in water pollution monitoring

Contemporarily, the quick development of an abundant amount of big data analysis technologies has brought great convenience to individuals' everyday existence. In terms of environmental protection, especially water pollution monitoring, this technological progress is particularly critical. As the global demand for clean water resources grows and global industrialization intensifies pollution of the water environment, the adoption of advanced data analysis technologies has become critical. Among the vast array of machine learning architectures, three particularly stand out due to their significance and widespread adoption: the artificial neural network (ANN), which serves as a foundational pillar in understanding complex data patterns; the multi-layer perceptron neural network (MLPNN), a sophisticated evolution that allows for deeper computations and learning; and the adaptive neuro-fuzzy inference system (ANFIS), which brilliantly combines neural and fuzzy logic principles for intricate problem solving. These models not only have high accuracy due to their wide application, but they still have their own limitations. This article aims to introduce the methods, basic principles, and application scenarios of these models. In addition, this article also compares the advantages and limitations of these machine learning models, thereby providing some new ideas for future improvements and innovations in model algorithms, application scenarios, and integration.

Evaluating large language models for annotating proteins.

In UniProtKB, up to date, there are more than 251 million proteins deposited. However, only 0.25% have been annotated with one of the more than 15000 possible Pfam family domains. The current annotation protocol integrates knowledge from manually curated family domains, obtained using sequence alignments and hidden Markov models. This approach has been successful for automatically growing the Pfam annotations, however at a low rate in comparison to protein discovery. Just a few years ago, deep learning models were proposed for automatic Pfam annotation. However, these models demand a considerable amount of training data, which can be a challenge with poorly populated families. To address this issue, we propose and evaluate here a novel protocol based on transfer learningṪhis requires the use of protein large language models (LLMs), trained with self-supervision on big unnanotated datasets in order to obtain sequence embeddings. Then, the embeddings can be used with supervised learning on a small and annotated dataset for a specialized task. In this protocol we have evaluated several cutting-edge protein LLMs together with machine learning architectures to improve the actual prediction of protein domain annotations. Results are significatively better than state-of-the-art for protein families classification, reducing the prediction error by an impressive 60% compared to standard methods. We explain how LLMs embeddings can be used for protein annotation in a concrete and easy way, and provide the pipeline in a github repo. Full source code and data are available at https://github.com/sinc-lab/llm4pfam.

Redefining the Laparoscopic Spatial Sense: AI-Based Intra- and Postoperative Measurement from Stereoimages

A significant challenge in image-guided surgery is the accurate measurement task of relevant structures such as vessel segments, resection margins, or bowel lengths. While this task is an essential component of many surgeries, it involves substantial human effort and is prone to inaccuracies. In this paper, we develop a novel human-AI-based method for laparoscopic measurements utilizing stereo vision that has been guided by practicing surgeons. Based on a holistic qualitative requirements analysis, this work proposes a comprehensive measurement method, which comprises state-of-the-art machine learning architectures, such as RAFT-Stereo and YOLOv8. The developed method is assessed in various realistic experimental evaluation environments. Our results outline the potential of our method achieving high accuracies in distance measurements with errors below 1 mm. Furthermore, on-surface measurements demonstrate robustness when applied in challenging environments with textureless regions. Overall, by addressing the inherent challenges of image-guided surgery, we lay the foundation for a more robust and accurate solution for intra- and postoperative measurements, enabling more precise, safe, and efficient surgical procedures.