ML-based Systems Research Articles

Evaluating the Cybersecurity Risk of Real-world, Machine Learning Production Systems

Although cyberattacks on machine learning (ML) production systems can be harmful, today, security practitioners are ill-equipped, lacking methodologies and tactical tools that would allow them to analyze the security risks of their ML-based systems. In this article, we perform a comprehensive threat analysis of ML production systems. In this analysis, we follow the ontology presented by NIST for evaluating enterprise network security risk and apply it to ML-based production systems. Specifically, we (1) enumerate the assets of a typical ML production system, (2) describe the threat model (i.e., potential adversaries, their capabilities, and their main goal), (3) identify the various threats to ML systems, and (4) review a large number of attacks, demonstrated in previous studies, which can realize these threats. To quantify the risk posed by adversarial machine learning (AML) threat, we introduce a novel scoring system that assigns a severity score to different AML attacks. The proposed scoring system utilizes the analytic hierarchy process (AHP) for ranking—with the assistance of security experts—various attributes of the attacks. Finally, we developed an extension to the MulVAL attack graph generation and analysis framework to incorporate cyberattacks on ML production systems. Using this extension, security practitioners can apply attack graph analysis methods in environments that include ML components thus providing security practitioners with a methodological and practical tool for both evaluating the impact and quantifying the risk of a cyberattack targeting ML production systems.

Bias and Discrimination in Ml-Based Systems of Administrative Decision-Making and Support

Bias and Discrimination in Ml-Based Systems of Administrative Decision-Making and Support

Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic Conflict Detection Predictor.

Designing and developing artificial intelligence (AI)-based systems that can be trusted justifiably is one of the main issues aviation must face in the coming years. European Union Aviation Safety Agency (EASA) has developed a user guide that could be potentially transformed as means of compliance for future AI-based regulation. Designers and developers must understand how the learning assurance process of any machine learning (ML) model impacts trust. ML is a narrow branch of AI that uses statistical models to perform predictions. This work deals with the learning assurance process for ML-based systems in the field of air traffic control. A conflict detection tool has been developed to identify separation infringements among aircraft pairs, and the ML algorithm used for classification and regression was extreme gradient boosting. This paper analyses the validity and adaptability of EASA W-shaped methodology for ML-based systems. The results have identified the lack of the EASA W-shaped methodology in time-dependent analysis, by showing how time can impact ML algorithms designed in the case where no time requirements are considered. Another meaningful conclusion is, for systems that depend highly on when the prediction is made, classification and regression metrics cannot be one-size-fits-all because they vary over time.

The Role of Explainability in Assuring Safety of Machine Learning in Healthcare

Established approaches to assuring safety-critical systems and software are difficult to apply to systems employing ML where there is no clear, pre-defined specification against which to assess validity. This problem is exacerbated by the “opaque” nature of ML where the learnt model is not amenable to human scrutiny. Explainable AI (XAI) methods have been proposed to tackle this issue by producing human-interpretable representations of ML models which can help users to gain confidence and build trust in the ML system. However, little work explicitly investigates the role of explainability for safety assurance in the context of ML development. This paper identifies ways in which XAI methods can contribute to safety assurance of ML-based systems. It then uses a concrete ML-based clinical decision support system, concerning weaning of patients from mechanical ventilation, to demonstrate how XAI methods can be employed to produce evidence to support safety assurance. The results are also represented in a safety argument to show where, and in what way, XAI methods can contribute to a safety case. Overall, we conclude that XAI methods have a valuable role in safety assurance of ML-based systems in healthcare but that they are not sufficient in themselves to assure safety.

Amortized Generation of Sequential Algorithmic Recourses for Black-Box Models

Explainable machine learning (ML) has gained traction in recent years due to the increasing adoption of ML-based systems in many sectors. Algorithmic Recourses (ARs) provide "what if" feedback of the form "if an input datapoint were x' instead of x, then an ML-based system's output would be y' instead of y." Recourses are attractive due to their actionable feedback, amenability to existing legal frameworks, and fidelity to the underlying ML model. Yet, current recourse approaches are single shot that is, they assume x can change to x' in a single time period. We propose a novel stochastic-control-based approach that generates sequential recourses, that is, recourses that allow x to move stochastically and sequentially across intermediate states to a final state x'. Our approach is model agnostic and black box. Furthermore, the calculation of recourses is amortized such that once trained, it applies to multiple datapoints without the need for re-optimization. In addition to these primary characteristics, our approach admits optional desiderata such as adherence to the data manifold, respect for causal relations, and sparsity identified by past research as desirable properties of recourses. We evaluate our approach using three real-world datasets and show successful generation of sequential recourses that respect other recourse desiderata.

Interpretable Model for Collaborative Filtering Using an Extended Latent Dirichlet Allocation Approach

With the increasing use of AI and ML-based systems, interpretability is becoming an increasingly important issue to ensure user trust and safety. This also applies to the area of recommender systems, where methods based on matrix factorization (MF) are among the most popular methods for collaborative filtering tasks with implicit feedback. Despite their simplicity, the latent factors of users and items lack interpretability in the case of the effective, unconstrained MF-based methods. In this work, we propose an extended latent Dirichlet Allocation model (LDAext) that has interpretable parameters such as user cohorts of item preferences and the affiliation of a user with different cohorts. We prove a theorem on how to transform the factors of an unconstrained MF model into the parameters of LDAext. Using this theoretical connection, we train an MF model on different real-world data sets, transform the latent factors into the parameters of LDAext and test their interpretation in several experiments for plausibility. Our experiments confirm the interpretability of the transformed parameters and thus demonstrate the usefulness of our proposed approach.

Federated Learning for Healthcare: Systematic Review and Architecture Proposal

The use of machine learning (ML) with electronic health records (EHR) is growing in popularity as a means to extract knowledge that can improve the decision-making process in healthcare. Such methods require training of high-quality learning models based on diverse and comprehensive datasets, which are hard to obtain due to the sensitive nature of medical data from patients. In this context, federated learning (FL) is a methodology that enables the distributed training of machine learning models with remotely hosted datasets without the need to accumulate data and, therefore, compromise it. FL is a promising solution to improve ML-based systems, better aligning them to regulatory requirements, improving trustworthiness and data sovereignty. However, many open questions must be addressed before the use of FL becomes widespread. This article aims at presenting a systematic literature review on current research about FL in the context of EHR data for healthcare applications. Our analysis highlights the main research topics, proposed solutions, case studies, and respective ML methods. Furthermore, the article discusses a general architecture for FL applied to healthcare data based on the main insights obtained from the literature review. The collected literature corpus indicates that there is extensive research on the privacy and confidentiality aspects of training data and model sharing, which is expected given the sensitive nature of medical data. Studies also explore improvements to the aggregation mechanisms required to generate the learning model from distributed contributions and case studies with different types of medical data.

Accuracy and Interpretability: Struggling with the Epistemic Foundations of Machine Learning-Generated Medical Information and Their Practical Implications for the Doctor-Patient Relationship

The initial successes in recent years in harnessing machine learning (ML) technologies to improve medical practice and benefit patients have attracted attention in a wide range of healthcare fields. Particularly, it should be achieved by providing automated decision recommendations to the treating clinician. Some hopes placed in such ML-based systems for healthcare, however, seem to be unwarranted, at least partially because of their inherent lack of transparency, although their results seem convincing in accuracy and reliability. Skepticism arises when the physician as the agent responsible for the implementation of diagnosis, therapy, and care is unable to access the generation of findings and recommendations. There is widespread agreement that, generally, a complete traceability is preferable to opaque recommendations; however, there are differences about addressing ML-based systems whose functioning seems to remain opaque to some degree—even if so-called explicable or interpretable systems gain increasing amounts of interest. This essay approaches the epistemic foundations of ML-generated information specifically and medical knowledge generally to advocate differentiations of decision-making situations in clinical contexts regarding their necessary depth of insight into the process of information generation. Empirically accurate or reliable outcomes are sufficient for some decision situations in healthcare, whereas other clinical decisions require extensive insight into ML-generated outcomes because of their inherently normative implications.

Abstract LB016: Deep learning identifies pathobiological features within H&E images associated with genomic alterations and progression on anti-PD(L)1 in HUDSON, an AstraZeneca-sponsored Phase II clinical trial

Abstract Introduction Machine learning (ML) models offer the potential to provide rich, quantitative characterizations of the tumor and tumor micro-environment (TME). Here we deployed a machine learning-based approach to the analysis of H&E images from HUDSON (NCT03334617), an AstraZeneca Phase II Platform clinical trial, to identify and quantify cellular composition and tissue architecture features in the TME that are associated with genomic alterations and time to progression on anti-PD(L)1 therapies. Methods PathAI previously trained ML models on non-small cell lung carcinoma (NSCLC) samples from commercial and clinical datasets to identify cell types and tissue regions within the TME. With no additional training, the models were deployed on 169 digitized whole slide images (WSIs) of H&E-stained biopsies from an international, multi-site AstraZeneca-sponsored Phase II clinical trial of novel anti-cancer agents in subjects with metastatic NSCLC. Biopsies were across multiple body sites, and taken both pre- and post-checkpoint progression. ML models generated human interpretable features (HIFs) that characterize the cell composition and tissue architecture from each biopsied sample. HIFs from baseline samples that met minimum image quality thresholds (n=89) were clustered to reduce redundancy and were tested for association with weeks to progression on anti-PD(L)1 therapy using Cox regression analysis. Results The PathAI ML models were successfully deployed on WSIs from the HUDSON clinical trial. Following correction for biopsy timing and location, a total of 59 HIFs were found to be significantly associated (p <0.05) with weeks to progression on anti-PD(L)1 therapy, including features related to plasma cell infiltration, proportion of cancer cells, presence of macrophages and fibroblasts, and blood vessel compression. Features characterizing both plasma cells and blood vessels were also found to be significantly associated with any class I HLA locus loss of heterozygosity. Conclusions PathAI models were able to identify TME-associated features from WSIs from a Phase II clinical trial which were associated with therapy failure and genomic alterations. These results suggest the power of deploying pre-trained ML-based systems in a clinical trial setting to identify pathobiological features associated with tumor characteristics and time to progression from only H&E images. Citation Format: Laura Dillon, Marylens Hernandez, Ben Glass, Guillaume Chhor, Sara Hoffman, Varsha Chinnaobireddy, Sai Chowdary Gullapally, Kris Sachsenmeier, Andy Beck, Jason Hipp. Deep learning identifies pathobiological features within H&E images associated with genomic alterations and progression on anti-PD(L)1 in HUDSON, an AstraZeneca-sponsored Phase II clinical trial [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2021; 2021 Apr 10-15 and May 17-21. Philadelphia (PA): AACR; Cancer Res 2021;81(13_Suppl):Abstract nr LB016.

Abstract PO-072: Robust deployment of ML models quantifying the H&E tumor microenvironment in NSCLC subjects from an AstraZeneca-sponsored phase II clinical trial

Abstract Introduction Machine learning models offer the potential to provide rich, quantitative characterizations of the tumor and tumor micro-environment (TME); however, historically it has been difficult to generalize trained models to new sets of clinical trial samples from trials not used in training. Here we evaluate the ability to deploy a machine learning based model (ML Model) for the identification of non-small cell lung tissue regions and lymphocytes within the tumor and TME on H&E stained images from clinical trial samples with no additional model training. Methods The ML model was previously trained on both squamous cell carcinoma and lung adenocarcinoma non-small cell lung carcinoma (NSCLC) samples from commercial and clinical datasets. The ML model was deployed on an AstraZeneca-sponsored phase II clinical trial of novel anti-cancer agents in patients with metastatic NSCLC. In order to validate the predictions of lymphocytes from the H&E stained images, we established a reference dataset for manual vs digital concordance consisting of 300, 150 × 150-micron–sized “frames” sampled from the trial dataset, removing frames of inadequate tissue quality or with presence of artifacts. For each frame, we collected exhaustive annotations from 5 pathologists to produce quantitative estimates of lymphocytes. Altogether, 43,932 annotations were collected and used to compute pathologist consensus scores for each frame. These scores were then correlated with each individual pathologist (inter-reader agreement) and with the PathAI-derived automated scores for evaluation of manual vs digital agreement. Results The PathAI system was successfully deployed on 169 H&E stained images from the phase II clinical trial to exhaustively identify all tumor associated lymphocytes from each whole slide image. In total, PathAI classified 2,859,796 lymphocytes, with an average number of 16,922 lymphocytes per image. We used frames-based validation to determine the correlation between the automated scoring and consensus scoring from pathologists hand labeling individual lymphocytes within image frames. The PathAI platform showed strong correlation between reference-based consensus scores (r2 = 0.84, CI [0.80 – 0.87]) and the ML model, which was similar to the level of agreement achieved between individual pathologists (r2 = 0.80, CI [0.76 – 0.85]). Conclusions The PathAI system showed strong generalizability for the identification of lymphocytes within the tumor and TME from H&E stained images from NSCLC clinical trial samples. These results suggest the power of deploying ML-based systems broadly for the automated, single cell resolution characterization of disease pathology from clinical trial material. Citation Format: Ben Glass, Laura Dillon, Guillaume Chhor, Sara Hoffman, Varsha Chinnaobireddy, Sai Chowdary Gullapally, Andy Beck, Jason Hipp. Robust deployment of ML models quantifying the H&E tumor microenvironment in NSCLC subjects from an AstraZeneca-sponsored phase II clinical trial [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-072.

Bladder cancer in the time of machine learning: Intelligent tools for diagnosis and management.

Machine learning (ML) is the subfield of artificial intelligence (AI), born from the marriage between statistics and computer science, with the unique purpose of building prediction algorithms able to improve their performances by automatically learning from massive data sets. The availability of ever-growing computational power and highly evolved pattern recognition software has led to the spread of ML-based systems able to perform complex tasks in bioinformatics, medical imaging, and diagnostics. These intelligent tools could be the answer to the unmet need for non-invasive and patient-tailored instruments for the diagnosis and management of bladder cancer (BC), which are still based on old technologies and unchanged nomograms. We reviewed the most significant evidence on ML in the diagnosis, prognosis, and management of bladder cancer, to find out if these intelligent technologies are ready to be introduced into the daily clinical practice of the urologist.

Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions

Background: Developing and maintaining large scale machine learning (ML) based software systems in an industrial setting is challenging. There are no well-established development guidelines, but the literature contains reports on how companies develop and maintain deployed ML-based software systems. Objective: This study aims to survey the literature related to development and maintenance of large scale ML-based systems in industrial settings in order to provide a synthesis of the challenges that practitioners face. In addition, we identify solutions used to address some of these challenges. Method: A systematic literature review was conducted and we identified 72 papers related to development and maintenance of large scale ML-based software systems in industrial settings. The selected articles were qualitatively analyzed by extracting challenges and solutions. The challenges and solutions were thematically synthesized into four quality attributes: adaptability, scalability, safety and privacy. The analysis was done in relation to ML workflow, i.e. data acquisition, training, evaluation, and deployment. Results: We identified a total of 23 challenges and 8 solutions related to development and maintenance of large scale ML-based software systems in industrial settings including six different domains. Challenges were most often reported in relation to adaptability and scalability. Safety and privacy challenges had the least reported solutions. Conclusion: The development and maintenance on large-scale ML-based systems in industrial settings introduce new challenges specific for ML, and for the known challenges characteristic for these types of systems, require new methods in overcoming the challenges. The identified challenges highlight important concerns in ML system development practice and the lack of solutions point to directions for future research.

Bridging the Gap between ISO 26262 and Machine Learning: A Survey of Techniques for Developing Confidence in Machine Learning Systems

<div class="section abstract"><div class="htmlview paragraph">Machine Learning (ML) based technologies are increasingly being used to fulfill safety-critical functions in autonomous and advanced driver assistance systems (ADAS). This change has been spurred by recent developments in ML and Artificial Intelligence techniques as well as rapid growth of computing power. However, demonstrating that ML-based systems achieve the necessary level of safety integrity remains a challenge. Current research and development work focused on establishing safe operation of ML-based systems presents individual techniques that might be used to gain confidence in these systems. As a result, there is minimal guidance for supporting a safety standard such as ISO 26262 - Road Vehicles - Functional Safety, to enable the development of ML-based systems. This paper presents a survey of recent ML literature to identify techniques and methods that can contribute to meeting ISO 26262 requirements. The surveyed literature is mapped onto the system development lifecycle V-model and the applicability of individual techniques and methods are discussed for each major phase of development.</div></div>

On testing machine learning programs

Nowadays, we are witnessing a wide adoption of Machine learning (ML) models in many software systems. They are even being tested in safety-critical systems, thanks to recent breakthroughs in deep learning and reinforcement learning. Many people are now interacting with systems based on ML every day, e.g., voice recognition systems used by virtual personal assistants like Amazon Alexa or Google Home. As the field of ML continues to grow, we are likely to witness transformative advances in a wide range of areas, from finance, energy, to health and transportation. Given this growing importance of ML-based systems in our daily life, it is becoming utterly important to ensure their reliability. Recently, software researchers have started adapting concepts from the software testing domain (e.g., code coverage, mutation testing, or property-based testing) to help ML engineers detect and correct faults in ML programs. This paper reviews current existing testing practices for ML programs. First, we identify and explain challenges that should be addressed when testing ML programs. Next, we report existing solutions found in the literature for testing ML programs. Finally, we identify gaps in the literature related to the testing of ML programs and make recommendations of future research directions for the scientific community. We hope that this comprehensive review of software testing practices will help ML engineers identify the right approach to improve the reliability of their ML-based systems. We also hope that the research community will act on our proposed research directions to advance the state of the art of testing for ML programs.

Machine Learning for Quantitative Finance Applications: A Survey

The analysis of financial data represents a challenge that researchers had to deal with. The rethinking of the basis of financial markets has led to an urgent demand for developing innovative models to understand financial assets. In the past few decades, researchers have proposed several systems based on traditional approaches, such as autoregressive integrated moving average (ARIMA) and the exponential smoothing model, in order to devise an accurate data representation. Despite their efficacy, the existing works face some drawbacks due to poor performance when managing a large amount of data with intrinsic complexity, high dimensionality and casual dynamicity. Furthermore, these approaches are not suitable for understanding hidden relationships (dependencies) between data. This paper proposes a review of some of the most significant works providing an exhaustive overview of recent machine learning (ML) techniques in the field of quantitative finance showing that these methods outperform traditional approaches. Finally, the paper also presents comparative studies about the effectiveness of several ML-based systems.

Viability Assessment of a Cross-Tokamak AUG-JET Disruption Predictor

Models that apply machine learning (ML) techniques for disruption prediction have improved detection rates and warning times in JET and other tokamaks. However, these models require an already stored database to develop them. Therefore, a significant problem arises at the time of training ML-based systems for ITER. To tackle this problem, this work computes a genetic algorithm–optimized predictor inspired by a previous study using initially only ASDEX-Upgrade (AUG) data and tested with the wide database of JET. This smaller-to-larger tokamak approach pursues the future extrapolation of this technique to ITER. The outcomes of direct application of a cross predictor resulted in 30.03% false alarms and more than 42% premature alarms, which indicates the need for different input parameters or at least some information about the target device to achieve reasonable performance.In a second approach, a new model was created with the AUG database plus one disruptive and one nondisruptive pulse of JET. The final cross predictions (over the chronologically first 564 shots after training, 52 of them were disruptive) reached 100% of total detected disruptions (all of them with anticipation times up to 10 ms). The false alarms were 7.42%. The results decayed at the time newer shots were tested. This aging effect is a known phenomenon, and it can be tackled by periodic retraining of the system. As proof of principle, a final predictor was created in an adaptive approach, obtaining in the following 1000 pulses (52 of them disruptive) 91.75% detections with at least 10 ms of warning times and less than 1% false alarms.