ML-based Systems Research Articles

Network anomaly detection and security defense technology based on machine learning: A review

Robust solutions are essential for protecting complex network systems in the constantly changing cybersecurity scenario. This investigation examines the role of machine learning (ML) in improving the safety of digital infrastructure by examining network anomaly detection and security defense. We evaluate the effectiveness of key ML techniques, including deep learning (DL), ensemble methods, and supervised and unsupervised learning, in recognizing abnormalities and reducing dangers. Real-time monitoring and adaptive reactions are emphasized in the analysis of ML-based systems integration into comprehensive security frameworks. This section also addresses challenges like interpretability, harmful assaults, and model resilience. Real-world case examples emphasize the importance of ML in the advancement of network security. The paper continues with a discussion of ML's future role in the optimization of security measures and the mitigation of emergent threats.

Enhancing cardiovascular risk assessment with advanced data balancing and domain knowledge-driven explainability

In medical risk prediction, such as predicting heart disease, machine learning (ML) classifiers must achieve high accuracy, precision, and recall to minimize the chances of incorrect diagnoses or treatment recommendations. However, real-world datasets often have imbalanced data, which can affect classifier performance. Traditional data balancing methods can lead to overfitting and underfitting, making it difficult to identify potential health risks accurately. Early prediction of heart attacks is of paramount importance, and researchers have developed ML-based systems to address this problem. However, much of the existing ML research is based on a single dataset, often ignoring performance evaluation across multiple datasets. As the demand for interpretable ML models grows, model interpretability becomes central to revealing insights and feature effects within predictive models. To address these challenges, we present a novel data balancing technique that uses a divide-and-conquer strategy with the K-Means clustering algorithm to segment the dataset. The performance of our approach is highlighted through comparisons with established techniques, which demonstrate the superiority of our proposed method. To address the challenge of inter-dataset discrepancies, we use two different datasets. Our holistic pipeline, strengthened by the innovative balancing technique, effectively addresses performance discrepancies, culminating in a significant improvement from 81% to 90%. Furthermore, through advanced statistical analysis, it has been determined that the 95% confidence interval for the AUC metric of our method ranges from 0.8187 to 0.8411. This observation serves to underscore the consistency and reliability of our approach, demonstrating its ability to achieve high performance across a range of scenarios. Incorporating Explainable AI (XAI), we examine the feature rankings and their contributions within the best performing Random Forest model. While the domain expert feedback is consistent with the explanatory power of XAI, some differences remain. Nevertheless, a remarkable convergence in feature ranking and weighting is observed, bridging the insights from XAI tools and domain expert perspectives.

Toward Sociotechnical AI: Mapping Vulnerabilities for Machine Learning in Context

This paper provides an empirical and conceptual account on seeing machine learning models as part of a sociotechnical system to identify relevant vulnerabilities emerging in the context of use. As ML is increasingly adopted in socially sensitive and safety-critical domains, many ML applications end up not delivering on their promises, and contributing to new forms of algorithmic harm. There is still a lack of empirical insights as well as conceptual tools and frameworks to properly understand and design for the impact of ML models in their sociotechnical context. In this paper, we follow a design science research approach to work towards such insights and tools. We center our study in the financial industry, where we first empirically map recently emerging MLOps practices to govern ML applications, and corroborate our insights with recent literature. We then perform an integrative literature research to identify a long list of vulnerabilities that emerge in the sociotechnical context of ML applications, and we theorize these along eight dimensions. We then perform semi-structured interviews in two real-world use cases and across a broad set of relevant actors and organizations, to validate the conceptual dimensions and identify challenges to address sociotechnical vulnerabilities in the design and governance of ML-based systems. The paper proposes a set of guidelines to proactively and integrally address both the dimensions of sociotechnical vulnerability, as well as the challenges identified in the empirical use case research, in the organization of MLOps practices.

Optimal hybrid classification model for event recommendation system

There is a growing need for recommender systems and other ML-based systems as an abundance of data is now available across all industries. Various industries are currently using recommender systems in slightly different ways. These programs utilize algorithms to propose appropriate products to consumers based on their prior choices and interactions. Moreover, Systems for recommending events to users suggest pertinent happenings that they might find interesting. As opposed to an object recommender that suggests books or movies; event-based recommender systems typically require distinct algorithms. A developed event recommendation method is introduced which includes two stages: feature extraction and recommendation. In stage, I, a Set of features like personal willingness, community willingness, informative content, edge weight, and node interest degree are extracted. Stage II of the event recommendation system performs a hybrid classification by combining LSTM and CNN. In the LSTM classifier, optimal tuning is done by Improvised Cat and Mouse optimization (ICMO) algorithm. The results of the ICMO technique at an 80% training percentage have the maximum sensitivity value of 95.19%, whereas those of the existing approaches SSA, DINGO, BOA, and CMBO have values of 93.89%, 93.35%, 92.36%, and 92.24%. Finally, the best result is then determined by evaluating the whole performance.

Machine learning-supported manufacturing: a review and directions for future research

ABSTRACT The evolution of manufacturing systems toward Industry 4.0 and 5.0 paradigms has pushed the diffusion of Machine Learning (ML) in this field. As the number of articles using ML to support manufacturing functions is expanding tremendously, the main objective of this review article is to provide a comprehensive and updated overview of these applications. 114 journal articles have been collected, analysed, and classified in terms of supervision approaches, function, ML algorithm, data inputs and outputs, and application domain. The findings show the fragmentation of the field and that most of the ML-based systems address limited objectives. Some inputs and outputs of the analysed support tools are shared across the reviewed contributions, and their possible combinations have been outlined. The advantages, limitations, and research opportunities of ML support in manufacturing are discussed. The paper outlines that the excessive specialization of the reviewed applications could be overcome by increasing the diffusion of transfer learning in the manufacturing domain.

An overview of key trustworthiness attributes and KPIs for trusted ML-based systems engineering

An overview of key trustworthiness attributes and KPIs for trusted ML-based systems engineering

Toward a safe MLOps process for the continuous development and safety assurance of ML-based systems in the railway domain

Toward a safe MLOps process for the continuous development and safety assurance of ML-based systems in the railway domain

A federated learning aided system for classifying cervical cancer using PAP-SMEAR images

Cervical cancer is a significant contributor to female mortality on a global scale, especially in low-income countries where effective screening programs for the detection and treatment of precancerous conditions are lacking. Classification of pap-smear test cervical cell images is crucial as it gives essential information for the diagnosis of malignant or precancerous lesions and thus helps in providing a proper diagnosis. Most of the existing methods require accumulating pap-smear test images of all patients in a centralized location for classification purposes. However, this procedure may hamper the privacy of patient data and create data ownership issues. In this study, different convolutional neural network-based federated learning architectures are introduced to achieve both the objectives of accurate image classification and data privacy in three different experimental settings. In the proposed system, the updates of the locally trained models get aggregated with an initially untrained global model in order to increase its performance. In traditional ML-based systems, the more the train data, the more efficiently the model performs, but in the proposed system, clients can participate remotely to train a robust model even with the disadvantage of possessing a small dataset. The proposed CNN-based FL architecture showed test accuracy of 94.36% and 78.4% in an IID (Independent and Identically Distributed) and a non-IID setting respectively. Thus multiple hospitals across different countries can use the proposed system to train their local models with their private dataset without sharing it centrally, which eventually will help to build the central model of federated learning architecture with diverse datasets.

How AI-Based Systems Can Induce Reflections: The Case of AI-Augmented Diagnostic Work

This paper addresses a thus-far neglected dimension in human-artificial intelligence (AI) augmentation: machine-induced reflections. By establishing a grounded theoretical-informed model of machine-induced reflection, we contribute to the ongoing discussion in information systems (IS) regarding AI and research on reflection theories. In our multistage study, physicians used a machine learning-based (ML) clinical decision support system (CDSS) to see if and how this interaction can stimulate reflective practice in the context of an X-ray diagnosis task. By analyzing verbal protocols, performance metrics, and survey data, we developed an integrative theoretical foundation to explain how ML-based systems can help stimulate reflective practice. Individuals engage in more critical or shallower modes depending on whether they perceive a conflict or agreement with these CDSS systems, which in turn leads to different levels of reflection depth. By uncovering the process of machine-induced reflections, we offer IS research a different perspective on how such AI-based systems can help individuals become more reflective, and consequently more effective, professionals. This perspective stands in stark contrast to the traditional, efficiency-focused view of ML-based decision support systems and also enriches theories on human-AI augmentation.

Cyber threat assessment of machine learning driven autonomous control systems of nuclear power plants

Advanced cyber-attacks against critical infrastructure and the energy sector are becoming more common. With the invention of autonomous control systems (ACS) within advanced nuclear reactor designs, system designers, reactor operators, and regulators must consider cybersecurity during the design and operational phases. This article provides a cyber threat assessment of machine learning (ML)-based digital twinning (DT) technologies in the context of advanced reactor ACS. A cyber–physical testbed was created to emulate nuclear reactor digital instrumentation and controls (I&C) and act as a basis for the ACS. The ACS was designed as two plant-level DTs predicting reactor malfunctions and determining control actions and two component-level DTs responsible for classifying component states and forecasting component inputs and outputs (I/O). Two duplicate ACS designs– one using a traditional ML framework and one using an automated ML (AutoML) framework– were created and tested against cyber-attacks on training data, real-time process data, and ML model architectures to determine their respective qualitative cyber-risk in terms of likelihood and impact. Both frameworks showed similar cyber-resilience against training, real-time, and ML architecture attacks, proving that neither is inherently more secure. Recommended safeguard and security measures are posed to system designers, reactor operators, and regulators to maintain the cybersecurity of ML-based DT technologies such as ACS, prompting a holistic view of shared responsibility for maintaining cyber-secure ML-based systems.

Methodology of real-time 3D point cloud mapping with UAV lidar

Accurate and timely availability of LiDAR data is vital in some cases. To facilitate monitoring of any environmental changes, LiDAR systems can be designed, and carried by UAV platforms that can take off without major preparation. In this study, the methodology of the real-time LiDAR mapping system was developed in the laboratory. The designed system shortens the target-based flight planning and post-flight data processing. In this system, the data is taken instantly and thus the change in the mapping area can be detected quickly. The simulation system, produce 3D point cloud, and data was stored in a database for later analysis. The 3D visualization of the data obtained from our developed UAV-LiDAR system was carried out with a platform-independent interface designed as web-based. The X3D file format used in the study to produce 3D point data provide an infrastructure for AI and ML-based systems in identifying urban objects in systems containing big data such as LiDAR.

Truthful meta-explanations for local interpretability of machine learning models

Automated Machine Learning-based systems’ integration into a wide range of tasks has expanded as a result of their performance and speed. Although there are numerous advantages to employing ML-based systems, if they are not interpretable, they should not be used in critical or high-risk applications. To address this issue, researchers and businesses have been focusing on finding ways to improve the explainability of complex ML systems, and several such methods have been developed. Indeed, there are so many developed techniques that it is difficult for practitioners to choose the best among them for their applications, even when using evaluation metrics. As a result, the demand for a selection tool, a meta-explanation technique based on a high-quality evaluation metric, is apparent. In this paper, we present a local meta-explanation technique which builds on top of the truthfulness metric, which is a faithfulness-based metric. We demonstrate the effectiveness of both the technique and the metric by concretely defining all the concepts and through experimentation.

Opening a New Era with Machine Learning in Financial Services? Forecasting Corporate Credit Ratings Based on Annual Financial Statements

Corporate credit ratings provide multiple strategic, financial, and managerial benefits for decision-makers. Therefore, it is essential to have accurate and up-to-date ratings to continuously monitor companies’ financial situations when making financial credit decisions. Machine learning (ML)-based internal models can be used for the assessment of companies’ financial situations using annual statements. Particularly, it is necessary to check whether these ML models achieve better results compared to statistical methods. Due to the multi-class classification problem when forecasting corporate credit ratings, the development, monitoring, and maintenance of ML-based systems are more challenging compared to simple classifications. This problem becomes even more complex due to the required coordination with financial regulators (e.g., OECD, EBA, BaFin, etc.). Furthermore, the ML models must be updated regularly due to the periodic nature of annual statements as a dataset. To address the problem of the limited dataset, multiple sampling strategies and machine learning algorithms can be combined for accurate and up-to-date forecasting of credit ratings. This paper provides various implications for ML-based forecasting of credit ratings and presents an approach for combining sampling strategies and ML techniques. It also provides design recommendations for ML-based services in the finance industry on how to fulfill the existing regulations.

Re-enacting machine learning practices to enquire into the moral issues they pose

As the number of ethical incidents associated with Machine Learning (ML) algorithms increases worldwide, many actors are seeking to produce technical and legal tools to regulate the professional practices associated with these technologies. However these tools, generally grounded either on lofty principles or on technical approaches, often fail at addressing the complexity of the moral issues that ML-based systems are triggering. They are mostly based on a ‘principled’ conception of morality where technical practices cannot be seen as more than mere means to be put at the service of more valuable moral ends. We argue that it is necessary to localise ethical debates within the complex entanglement of technical, legal and organisational entities from which ML moral issues stem. To expand the repertoire of the approaches through which these issues might be addressed, we designed and tested an interview protocol based on the re-enactment of data scientists’ daily ML practices. We asked them to recall and describe the crafting and choosing of algorithms. Then, our protocol added two reflexivity-fostering elements to the situation: technical tools to assess algorithms’ morality, based on incorporated ‘ethicality’ indicators; and a series of staged objections to the aforementioned technical solutions to ML moral issues, made by factitious actors inspired by the data scientists’ daily environment. We used this protocol to observe how ML data scientists uncover associations with multiple entities, to address moral issues from within the course of their technical practices. We thus reframe ML morality as an inquiry into the uncertain options that practitioners face in the heat of technical activities. We propose to institute moral enquiries both as a descriptive method serving to delineate alternative depictions of ML algorithms when they are affected by moral issues and as a transformative method to propagate situated critical technical practices within ML-building professional environments.

Artificial Intelligence Supporting the Training of Communication Skills in the Education of Health Care Professions: Scoping Review.

Communication is a crucial element of every health care profession, rendering communication skills training in all health care professions as being of great importance. Technological advances such as artificial intelligence (AI) and particularly machine learning (ML) may support this cause: it may provide students with an opportunity for easily accessible and readily available communication training. This scoping review aimed to summarize the status quo regarding the use of AI or ML in the acquisition of communication skills in academic health care professions. We conducted a comprehensive literature search across the PubMed, Scopus, Cochrane Library, Web of Science Core Collection, and CINAHL databases to identify articles that covered the use of AI or ML in communication skills training of undergraduate students pursuing health care profession education. Using an inductive approach, the included studies were organized into distinct categories. The specific characteristics of the studies, methods and techniques used by AI or ML applications, and main outcomes of the studies were evaluated. Furthermore, supporting and hindering factors in the use of AI and ML for communication skills training of health care professionals were outlined. The titles and abstracts of 385 studies were identified, of which 29 (7.5%) underwent full-text review. Of the 29 studies, based on the inclusion and exclusion criteria, 12 (3.1%) were included. The studies were organized into 3 distinct categories: studies using AI and ML for text analysis and information extraction, studies using AI and ML and virtual reality, and studies using AI and ML and the simulation of virtual patients, each within the academic training of the communication skills of health care professionals. Within these thematic domains, AI was also used for the provision of feedback. The motivation of the involved agents played a major role in the implementation process. Reported barriers to the use of AI and ML in communication skills training revolved around the lack of authenticity and limited natural flow of language exhibited by the AI- and ML-based virtual patient systems. Furthermore, the use of educational AI- and ML-based systems in communication skills training for health care professionals is currently limited to only a few cases, topics, and clinical domains. The use of AI and ML in communication skills training for health care professionals is clearly a growing and promising field with a potential to render training more cost-effective and less time-consuming. Furthermore, it may serve learners as an individualized and readily available exercise method. However, in most cases, the outlined applications and technical solutions are limited in terms of access, possible scenarios, the natural flow of a conversation, and authenticity. These issues still stand in the way of any widespread implementation ambitions.

Application of machine learning in predicting non-alcoholic fatty liver disease using anthropometric and body composition indices

Non-alcoholic fatty liver disease (NAFLD) is the most common chronic liver disease, which can progress from simple steatosis to advanced cirrhosis and hepatocellular carcinoma. Clinical diagnosis of NAFLD is crucial in the early stages of the disease. The main aim of this study was to apply machine learning (ML) methods to identify significant classifiers of NAFLD using body composition and anthropometric variables. A cross-sectional study was carried out among 513 individuals aged 13 years old or above in Iran. Anthropometric and body composition measurements were performed manually using body composition analyzer InBody 270. Hepatic steatosis and fibrosis were determined using a Fibroscan. ML methods including k-Nearest Neighbor (kNN), Support Vector Machine (SVM), Radial Basis Function (RBF) SVM, Gaussian Process (GP), Random Forest (RF), Neural Network (NN), Adaboost and Naïve Bayes were examined for model performance and to identify anthropometric and body composition predictors of fatty liver disease. RF generated the most accurate model for fatty liver (presence of any stage), steatosis stages and fibrosis stages with 82%, 52% and 57% accuracy, respectively. Abdomen circumference, waist circumference, chest circumference, trunk fat and body mass index were among the most important variables contributing to fatty liver disease. ML-based prediction of NAFLD using anthropometric and body composition data can assist clinicians in decision making. ML-based systems provide opportunities for NAFLD screening and early diagnosis, especially in population-level and remote areas.

Clinician-Facing AI in the Wild: Taking Stock of the Sociotechnical Challenges and Opportunities for HCI

Artificial Intelligence (AI) in medical applications holds great promise. However, the use of Machine Learning-based (ML) systems in clinical practice is still minimal. It is uniquely difficult to introduce clinician-facing ML-based systems in practice, which has been recognised in HCI and related fields. Recent publications have begun to address the sociotechnical challenges of designing, developing, and successfully deploying clinician-facing ML-based systems. We conducted a qualitative systematic review and provided answers to the question: “How can HCI researchers and practitioners contribute to the successful realisation of ML in medical practice?” We reviewed 25 eligible papers that investigated the real-world clinical implications of concrete clinician-facing ML-based systems. The main contributions of this systematic review are: (1) an overview of the technical aspects of ML innovation and their consequences for HCI researchers and practitioners; (2) a description of the different roles that ML-based systems can take in clinical settings; (3) a conceptualisation of the main activities of medical ML innovation processes; (4) identification of five sociotechnical interdependencies that emerge from medical ML innovation; and (5) implications for HCI researchers and practitioners on how to mitigate the sociotechnical challenges of medical ML innovation.

Ergo, SMIRK is safe: a safety case for a machine learning component in a pedestrian automatic emergency brake system

Integration of machine learning (ML) components in critical applications introduces novel challenges for software certification and verification. New safety standards and technical guidelines are under development to support the safety of ML-based systems, e.g., ISO 21448 SOTIF for the automotive domain and the Assurance of Machine Learning for use in Autonomous Systems (AMLAS) framework. SOTIF and AMLAS provide high-level guidance but the details must be chiseled out for each specific case. We initiated a research project with the goal to demonstrate a complete safety case for an ML component in an open automotive system. This paper reports results from an industry-academia collaboration on safety assurance of SMIRK, an ML-based pedestrian automatic emergency braking demonstrator running in an industry-grade simulator. We demonstrate an application of AMLAS on SMIRK for a minimalistic operational design domain, i.e., we share a complete safety case for its integrated ML-based component. Finally, we report lessons learned and provide both SMIRK and the safety case under an open-source license for the research community to reuse.

An interpretable machine learning approach to multimodal stress detection in a simulated office environment

Background and objective:Work-related stress affects a large part of today’s workforce and is known to have detrimental effects on physical and mental health. Continuous and unobtrusive stress detection may help prevent and reduce stress by providing personalised feedback and allowing for the development of just-in-time adaptive health interventions for stress management. Previous studies on stress detection in work environments have often struggled to adequately reflect real-world conditions in controlled laboratory experiments. To close this gap, in this paper, we present a machine learning methodology for stress detection based on multimodal data collected from unobtrusive sources in an experiment simulating a realistic group office environment (N=90). Methods:We derive mouse, keyboard and heart rate variability features to detect three levels of perceived stress, valence and arousal with support vector machines, random forests and gradient boosting models using 10-fold cross-validation. We interpret the contributions of features to the model predictions with SHapley Additive exPlanations (SHAP) value plots. Results:The gradient boosting models based on mouse and keyboard features obtained the highest average F1 scores of 0.625, 0.631 and 0.775 for the multiclass prediction of perceived stress, arousal and valence, respectively. Our results indicate that the combination of mouse and keyboard features may be better suited to detect stress in office environments than heart rate variability, despite physiological signal-based stress detection being more established in theory and research. The analysis of SHAP value plots shows that specific mouse movement and typing behaviours may characterise different levels of stress. Conclusions:Our study fills different methodological gaps in the research on the automated detection of stress in office environments, such as approximating real-life conditions in a laboratory and combining physiological and behavioural data sources. Implications for field studies on personalised, interpretable ML-based systems for the real-time detection of stress in real office environments are also discussed.

Enhancing System Performance through Objective Feature Scoring of Multiple Persons' Breathing Using Non-Contact RF Approach.

Breathing monitoring is an efficient way of human health sensing and predicting numerous diseases. Various contact and non-contact-based methods are discussed in the literature for breathing monitoring. Radio frequency (RF)-based breathing monitoring has recently gained enormous popularity among non-contact methods. This method eliminates privacy concerns and the need for users to carry a device. In addition, such methods can reduce stress on healthcare facilities by providing intelligent digital health technologies. These intelligent digital technologies utilize a machine learning (ML)-based system for classifying breathing abnormalities. Despite advances in ML-based systems, the increasing dimensionality of data poses a significant challenge, as unrelated features can significantly impact the developed system's performance. Optimal feature scoring may appear to be a viable solution to this problem, as it has the potential to improve system performance significantly. Initially, in this study, software-defined radio (SDR) and RF sensing techniques were used to develop a breathing monitoring system. Minute variations in wireless channel state information (CSI) due to breathing movement were used to detect breathing abnormalities in breathing patterns. Furthermore, ML algorithms intelligently classified breathing abnormalities in single and multiple-person scenarios. The results were validated by referencing a wearable sensor. Finally, optimal feature scoring was used to improve the developed system's performance in terms of accuracy, training time, and prediction speed. The results showed that optimal feature scoring can help achieve maximum accuracy of up to 93.8% and 91.7% for single-person and multi-person scenarios, respectively.