Large-scale Machine Learning Research Articles

Photonic neuromorphic architecture for tens-of-task lifelong learning.

Scalable, high-capacity, and low-power computing architecture is the primary assurance for increasingly manifold and large-scale machine learning tasks. Traditional electronic artificial agents by conventional power-hungry processors have faced the issues of energy and scaling walls, hindering them from the sustainable performance improvement and iterative multi-task learning. Referring to another modality of light, photonic computing has been progressively applied in high-efficient neuromorphic systems. Here, we innovate a reconfigurable lifelong-learning optical neural network (L2ONN), for highly-integrated tens-of-task machine intelligence with elaborated algorithm-hardware co-design. Benefiting from the inherent sparsity and parallelism in massive photonic connections, L2ONN learns each single task by adaptively activating sparse photonic neuron connections in the coherent light field, while incrementally acquiring expertise on various tasks by gradually enlarging the activation. The multi-task optical features are parallelly processed by multi-spectrum representations allocated with different wavelengths. Extensive evaluations on free-space and on-chip architectures confirm that for the first time, L2ONN avoided the catastrophic forgetting issue of photonic computing, owning versatile skills on challenging tens-of-tasks (vision classification, voice recognition, medical diagnosis, etc.) with a single model. Particularly, L2ONN achieves more than an order of magnitude higher efficiency than the representative electronic artificial neural networks, and 14× larger capacity than existing optical neural networks while maintaining competitive performance on each individual task. The proposed photonic neuromorphic architecture points out a new form of lifelong learning scheme, permitting terminal/edge AI systems with light-speed efficiency and unprecedented scalability.

Single-cell and whole-transcriptome sequencing of lymph node metastasis-related gene signature: A large-scale pan-cancer cohort, machine learning and experimental study: [RETRACTED

Lymph node metastasis (LNM) is an independent prognostic factor in numerous types of cancer. Therefore, a LNM-related gene-based nomogram may precisely predict survival and drug sensitivity, and reveal the mechanism underlying LNM. Gene sequencing profiles of pan-cancer data (33 cancer types) were acquired from The Cancer Genome Atlas UCSC Xena database. Patients were classified into primary (N = 10,071) and testing (N = 5,036) cohorts. The lymph node score (LNscore) was established via single-cell RNA sequencing, whole-transcriptome sequencing, machine learning, and Cox regression analyses. A novel prognosis model, formulated by incorporating the LNscore and clinical characteristics, was evaluated using the concordance index, calibration curve, and decision curve analysis. Moreover, patients were assigned into high- and low-risk groups according to the median LNscore. We investigated these two groups for survival prognosis, functional enrichment, immune infiltration, and drug sensitivity. In addition, we silenced and overexpressed insulin like growth factor 2 mRNA binding protein 2 (IGF2BP2). We also analyzed the behavior of breast cancer (BRCA) cells regarding lymphatic metastasis and lymphangiogenesis in vitro. IGF2BP2 stimulated the proliferation of BRCA cells via 5-Ethynyl-2'-deoxyuridine and Cell Counting Kit-8 experiments. A LNM-related set of 12 genes was identified and utilized to determine the LNscore. The concordance-index of both cohorts in the LNscore-based model was >0.7. The immune landscape revealed that the sensitivity to immunotherapy might be better in the high-risk group versus the low-risk group. In addition, we discovered that IGF2BP2 was overexpressed in BRCA tissues and significantly associated with poor survival. Functional analysis indicated that IGF2BP2 promoted BRCA cell migration and proliferation. Additionally, IGF2BP2 accelerated lymphatic metastasis and lymphangiogenesis in vivo. A novel LNscore-based model was established via comprehensive analysis of LNM-related genes. This model can accurately predict patient survival and drug sensitivity, and reveal the mechanism of LNM in the pan-cancer setting.

Tailoring Gradient Methods for Differentially Private Distributed Optimization

Decentralized optimization is gaining increased traction due to its widespread applications in large-scale machine learning and multi-agent systems. The same mechanism that enables its success, i.e., information sharing among participating agents, however, also leads to the disclosure of individual agents' private information, which is unacceptable when sensitive data are involved. As differential privacy is becoming a de facto standard for privacy preservation, recently results have emerged integrating differential privacy with distributed optimization. However, directly incorporating differential privacy design in existing distributed optimization approaches significantly compromises optimization accuracy. In this paper, we propose to redesign and tailor gradient methods for differentially-private distributed optimization, and propose two differential-privacy oriented gradient methods that can ensure both rigorous <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\epsilon$</tex-math></inline-formula> -differential privacy and optimality. The first algorithm is based on static-consensus based gradient methods, and the second algorithm is based on dynamic-consensus (gradient-tracking) based distributed optimization methods and, hence, is applicable to general directed interaction graph topologies. Both algorithms can simultaneously ensure almost sure convergence to an optimal solution and a finite privacy budget, even when the number of iterations goes to infinity. To our knowledge, this is the first time that both goals are achieved simultaneously. Numerical simulations using a distributed estimation problem and experimental results on a benchmark dataset confirm the effectiveness of the proposed approaches.

Beyond Gaia DR3: Tracing the [α/M] – [M/H] bimodality from the inner to the outer Milky Way disc with Gaia-RVS and convolutional neural networks

Context. In June 2022, Gaia DR3 provided the astronomy community with about one million spectra from the Radial Velocity Spectrometer (RVS) covering the CaII triplet region. In the next Gaia data releases, we anticipate the number of RVS spectra to successively increase from several 10 million spectra to eventually more than 200 million spectra. Thus, stellar spectra are projected to be produced on an ‘industrial scale’, with numbers well above those for current and anticipated ground-based surveys. However, one-third of the published spectra have 15 ≤ S /N ≤ 25 per pixel such that they pose problems for classical spectral analysis pipelines, and therefore, alternative ways to tap into these large datasets need to be devised. Aims. We aim to leverage the versatility and capabilities of machine learning techniques for supercharged stellar parametrisation by combining Gaia-RVS spectra with the full set of Gaia products and high-resolution, high-quality ground-based spectroscopic reference datasets. Methods. We developed a hybrid convolutional neural network (CNN) that combines the Gaia DR3 RVS spectra, photometry (G, G_BP, G_RP), parallaxes, and XP coefficients to derive atmospheric parameters (Teff, log(g) as well as overall [M/H]) and chemical abundances ([Fe/H] and [α/M]). We trained the CNN with a high-quality training sample based on APOGEE DR17 labels. Results. With this CNN, we derived homogeneous atmospheric parameters and abundances for 886 080 RVS stars that show remarkable precision and accuracy compared to external datasets (such as GALAH and asteroseismology). The CNN is robust against noise in the RVS data, and we derive very precise labels down to S/N =15. We managed to characterise the [α/M] - [M/H] bimodality from the inner regions to the outer parts of the Milky Way, which has never been done using RVS spectra or similar datasets. Conclusions. This work is the first to combine machine learning with such diverse datasets and paves the way for large-scale machine learning analysis of Gaia-RVS spectra from future data releases. Large, high-quality datasets can be optimally combined thanks to the CNN, thereby realising the full power of spectroscopy, astrometry, and photometry.

Large-Scale Data Processing and Machine Learning Analysis Model Based on Distributed Algorithm

The model of large-scale data processing and ML(machine learning) analysis based on DA(distributed algorithm) is a powerful computing method, which aims at processing huge data sets and performing efficient ML analysis. In this paper, a cluster topology driver module based on gradient switching and aggregate communication is designed, and its core goal is to adapt the distributed system to various underlying network topologies. By designing decentralized gradient exchange algorithm and aggregate communication framework, the parallel transmission ability of multi-interface network can be fully exerted, thus improving the model synchronization efficiency of ML task. The experimental results show that the cluster topology driver module can provide better performance than the existing methods in terms of training convergence, cluster scalability and communication overhead. Large-scale data processing and ML analysis model based on DA is widely used in processing massive data and realizing complex analysis tasks.

Hyperscaling in the Coherent Hyperspin Machine.

Classical and quantum systems are used to simulate the Ising Hamiltonian, an essential component in large-scale optimization and machine learning. However, as the system size increases, devices like quantum annealers and coherent Ising machines face an exponential drop in their success rate. Here, we introduce a novel approach involving high-dimensional embeddings of the Ising Hamiltonian and a technique called "dimensional annealing" to counteract the decrease in performance. This approach leads to an exponential improvement in the success rate and other performance metrics, slowing down the decline in performance as the system size grows. A thorough examination of convergence dynamics in high-performance computing validates the new methodology. Additionally, we suggest practical implementations using technologies like coherent Ising machines, all-optical systems, and hybrid digital systems. The proposed hyperscaling heuristics can also be applied to other quantum or classical Ising devices by adjusting parameters such as nonlinear gain, loss, and nonlocal couplings.

Distributed Machine Learning Workflow with PanDA and iDDS in LHC ATLAS

Machine Learning (ML) has become one of the important tools for High Energy Physics analysis. As the size of the dataset increases at the Large Hadron Collider (LHC), and at the same time the search spaces become bigger and bigger in order to exploit the physics potentials, more and more computing resources are required for processing these ML tasks. In addition, complex advanced ML workflows are developed in which one task may depend on the results of previous tasks. How to make use of vast distributed CPUs/GPUs in WLCG for these big complex ML tasks has become a popular research area. In this paper, we present our efforts enabling the execution of distributed ML workflows on the Production and Distributed Analysis (PanDA) system and intelligent Data Delivery Service (iDDS). First, we describe how PanDA and iDDS deal with large-scale ML workflows, including the implementation to process workloads on diverse and geographically distributed computing resources. Next, we report real-world use cases, such as HyperParameter Optimization, Monte Carlo Toy confidence limits calculation, and Active Learning. Finally, we conclude with future plans.

The impact of characteristic function of Shapley value mechanism in distributed machine learning environment for equipment diagnosis

The distributed machine learning environment with multiple data providers has recently gotten attention. By using this environment, a production equipment manufacturer and its customer companies can collaboratively perform large-scale machine learning for equipment diagnosis. To construct a high-performance machine learning model, the environment should compensate and reward each data provider for sensing, collecting, and cleansing relevant and informative data. Most previous studies use Shapley-value-based incentive mechanism for this purpose. However, they do not consider the impact of the characteristic function used to evaluate the Shapley value on the behavior of the data providers and the performance of the final machine learning model. This research evaluates the impact through evolutionary game simulation.

Large-Scale Machine Learning Analysis Reveals DNA Methylation and Gene Expression Response Signatures for Gemcitabine-Treated Pancreatic Cancer.

Background: Gemcitabine is a first-line chemotherapy for pancreatic adenocarcinoma (PAAD), but many PAAD patients do not respond to gemcitabine-containing treatments. Being able to predict such nonresponders would hence permit the undelayed administration of more promising treatments while sparing gemcitabine life-threatening side effects for those patients. Unfortunately, the few predictors of PAAD patient response to this drug are weak, none of them exploiting yet the power of machine learning (ML). Methods: Here, we applied ML to predict the response of PAAD patients to gemcitabine from the molecular profiles of their tumors. More concretely, we collected diverse molecular profiles of PAAD patient tumors along with the corresponding clinical data (gemcitabine responses and clinical features) from the Genomic Data Commons resource. From systematically combining 8 tumor profiles with 16 classification algorithms, each of the resulting 128 ML models was evaluated by multiple 10-fold cross-validations. Results: Only 7 of these 128 models were predictive, which underlines the importance of carrying out such a large-scale analysis to avoid missing the most predictive models. These were here random forest using 4 selected mRNAs [0.44 Matthews correlation coefficient (MCC), 0.785 receiver operating characteristic-area under the curve (ROC-AUC)] and XGBoost combining 12 DNA methylation probes (0.32 MCC, 0.697 ROC-AUC). By contrast, the hENT1 marker obtained much worse random-level performance (practically 0 MCC, 0.5 ROC-AUC). Despite not being trained to predict prognosis (overall and progression-free survival), these ML models were also able to anticipate this patient outcome. Conclusions: We release these promising ML models so that they can be evaluated prospectively on other gemcitabine-treated PAAD patients.

Investigating the error imbalance of large-scale machine learning potentials in catalysis

Removing calculations with surface reconstructions reduces the MAEs of the MLPs.

Federated Motor Imagery Classification for Privacy-Preserving Brain-Computer Interfaces.

Training an accurate classifier for EEG-based brain-computer interface (BCI) requires EEG data from a large number of users, whereas protecting their data privacy is a critical consideration. Federated learning (FL) is a promising solution to this challenge. This paper proposes Federated classification with local Batch-specific batch normalization and Sharpness-aware minimization (FedBS) for privacy protection in EEG-based motor imagery (MI) classification. FedBS utilizes local batch-specific batch normalization to reduce data discrepancies among different clients, and sharpness-aware minimization optimizer in local training to improve model generalization. Experiments on three public MI datasets using three popular deep learning models demonstrated that FedBS outperformed six state-of-the-art FL approaches. Remarkably, it also outperformed centralized training, which does not consider privacy protection at all. In summary, FedBS protects user EEG data privacy, enabling multiple BCI users to participate in large-scale machine learning model training, which in turn improves the BCI decoding accuracy.

Targeted Display Advertising using Machine Learning

This paper delves into the intricate challenges of problem formulation and data representation in the context of a large-scale machine learning system for targeted display advertising. Unlike traditional models, this system is not just conceptual but has been operational for years across thousands of advertising campaigns. Since obtaining ideal training data is cost-prohibitive, data is sourced from related domains and tasks and then adapted for the target task. The paper outlines the architecture of this multi-stage transfer learning system, emphasizing the problem formulation aspects. Extensive experiments demonstrate the value of each transfer stage. Real-world results with diverse advertising clients from various industries showcase the system's performance. The paper concludes with valuable insights gained from over half a decade of work on this complex, widely deployed machine learning system.

DPro-SM – A distributed framework for proactive straggler mitigation using LSTM

The recent advancement in deep learning with growth in big data and high-performance computing is Distributed Deep Learning. The rapid rise in the volume of data and network complexity has led to significant growth in DDL. Distribution of the network leads to high communication and computation among the nodes, which leads to high training time and lower accuracy. The primary reason for the delay in communication is the presence of straggler nodes which causes the bottleneck in communication. Due to the enormous volume of parameter transfer, Distributed Deep Learning's data parallelism incurs substantial communication costs. The newly developed model-parallel methods may minimize the communication effort; however, this results in load imbalance and severe straggler issues: the proposed model DPro-SM, a distributed framework for proactive straggler mitigation using LSTM in distributed deep learning. DPro-SM uses LSTM to predict the completion time of each worker and proactively allocates resources to reduce the overall training time. The results show that DPro-SM can significantly reduce the training time and improve the scalability and efficiency of large-scale machine learning tasks.

Towards Reliable Federated Learning Using Blockchain-Based Reverse Auctions and Reputation Incentives

In recent years, the explosion of big data has presented unparalleled opportunities for the advancement of machine learning (ML). However, the vast size and sensitive nature of these datasets present significant challenges in terms of privacy and security. Federated Learning has emerged as a promising solution that enables a group of participants to train ML models without compromising the confidentiality of their raw data. Despite its potential, traditional federated learning faces challenges such as the absence of participant incentives and audit mechanisms. Furthermore, these challenges become more significant when dealing with the scale and diversity of big data, making efficient and reliable federated learning a complex task. These limitations may compromise model quality due to potential malicious nodes. To address the above issues, this paper proposes a BlockChain-based Decentralized Federated Learning (BCD-FL) model. In BCD-FL, we design a smart contract approach based on the reverse auction-based incentive mechanism and a reputation mechanism to promote the participation of reliable and high-quality data owners. Theoretical analysis shows that the BCD-FL model satisfies several desirable properties, such as individual rationality, computational efficiency, budget balance, and truthfulness. In addition, experimental results also show that the proposed model enables more efficient federated learning and provides some level of protection against malicious nodes. Therefore, the BCD-FL model presents a potential solution to the challenges in federated learning and opens up new possibilities for achieving efficient large-scale machine learning.

Novel molecular classification and prognosis of papillary renal cell carcinoma based on a large-scale CRISPR-Cas9 screening and machine learning

Papillary renal cell carcinoma (PRCC) is a highly heterogeneous cancer, and PRCC patients with advanced/metastatic subgroup showed obviously shorter survival compared to other kinds of renal cell carcinomas. However, the molecular mechanism and prognostic predictors of PRCC remain unclear and are worth deep studying. The aim of this study is to identify novel molecular classification and construct a reliable prognostic model for PRCC. The expression data were retrieved from TCGA, GEO, GTEx and TARGET databases. CRISPR data was obtained from Depmap database. The key genes were selected by the intersection of CRISPR-Cas9 screening genes, differentially expressed genes, and genes with prognostic capacity in PRCC. The molecular classification was identified based on the key genes. Drug sensitivity, tumor microenvironment, somatic mutation, and survival were compared among the novel classification. A prognostic model utilizing multiple machine learning algorithms based on the key genes was developed and tested by independent external validation set. Our study identified three clusters (C1, C2 and C3) in PRCC based on 41 key genes. C2 had obviously higher expression of the key genes and lower survival than C1 and C3. Significant differences in drug sensitivity, tumor microenvironment, and mutation landscape have been observed among the three clusters. By utilizing 21 combinations of 9 machine learning algorithms, 9 out of 41 genes were chosen to construct a robust prognostic signature, which exhibited good prognostic ability. SERPINH1 was identified as a critical gene for its strong prognostic ability in PRCC by univariate and multiple Cox regression analyses. Quantitative real-time PCR and Western blot demonstrated that SERPINH1 mRNA and protein were highly expressed in PRCC cells compared with normal human renal cells. This study exhibited a new molecular classification and prognostic signature for PRCC, which may provide a potential biomarker and therapy target for PRCC patients.

Machine learning code snippets semantic classification

Program code has recently become a valuable active data source for training various data science models, from code classification to controlled code synthesis. Annotating code snippets play an essential role in such tasks. This article presents a novel approach that leverages CodeBERT, a powerful transformer-based model, to classify code snippets extracted from Code4ML automatically. Code4ML is a comprehensive machine learning code corpus compiled from Kaggle, a renowned data science competition platform. The corpus includes code snippets and information about the respective kernels and competitions, but it is limited in the quality of the tagged data, which is ~0.2%. Our method addresses the lack of labeled snippets for supervised model training by exploiting the internal ambiguity in particular labeled snippets where multiple class labels are combined. Using a specially designed algorithm, we effectively separate these ambiguous fragments, thereby expanding the pool of training data. This data augmentation approach greatly increases the amount of labeled data and improves the overall quality of the trained models. The experimental results demonstrate the prowess of the proposed code classifier, achieving an impressive F1 test score of ~89%. This achievement not only enhances the practicality of CodeBERT for classifying code snippets but also highlights the importance of enriching large-scale annotated machine learning code datasets such as Code4ML. With a significant increase in accurately annotated code snippets, Code4ML is becoming an even more valuable resource for learning and improving various data processing models.

Machine-guided discovery of a real-world rogue wave model.

Big data and large-scale machine learning have had a profound impact on science and engineering, particularly in fields focused on forecasting and prediction. Yet, it is still not clear how we can use the superior pattern-matching abilities of machine learning models for scientific discovery. This is because the goals of machine learning and science are generally not aligned. In addition to being accurate, scientific theories must also be causally consistent with the underlying physical process and allow for human analysis, reasoning, and manipulation to advance the field. In this paper, we present a case study on discovering a symbolic model for oceanic rogue waves from data using causal analysis, deep learning, parsimony-guided model selection, and symbolic regression. We train an artificial neural network on causal features from an extensive dataset of observations from wave buoys, while selecting for predictive performance and causal invariance. We apply symbolic regression to distill this black-box model into a mathematical equation that retains the neural network's predictive capabilities, while allowing for interpretation in the context of existing wave theory. The resulting model reproduces known behavior, generates well-calibrated probabilities, and achieves better predictive scores on unseen data than current theory. This showcases how machine learning can facilitate inductive scientific discovery and paves the way for more accurate rogue wave forecasting.

Advancing Federated Learning through Verifiable Computations and Homomorphic Encryption.

Federated learning, as one of the three main technical routes for privacy computing, has been widely studied and applied in both academia and industry. However, malicious nodes may tamper with the algorithm execution process or submit false learning results, which directly affects the performance of federated learning. In addition, learning nodes can easily obtain the global model. In practical applications, we would like to obtain the federated learning results only by the demand side. Unfortunately, no discussion on protecting the privacy of the global model is found in the existing research. As emerging cryptographic tools, the zero-knowledge virtual machine (ZKVM) and homomorphic encryption provide new ideas for the design of federated learning frameworks. We have introduced ZKVM for the first time, creating learning nodes as local computing provers. This provides execution integrity proofs for multi-class machine learning algorithms. Meanwhile, we discuss how to generate verifiable proofs for large-scale machine learning tasks under resource constraints. In addition, we implement the fully homomorphic encryption (FHE) scheme in ZKVM. We encrypt the model weights so that the federated learning nodes always collaborate in the ciphertext space. The real results can be obtained only after the demand side decrypts them using the private key. The innovativeness of this paper is demonstrated in the following aspects: 1. We introduce the ZKVM for the first time, which achieves zero-knowledge proofs (ZKP) for machine learning tasks with multiple classes and arbitrary scales. 2. We encrypt the global model, which protects the model privacy during local computation and transmission. 3. We propose and implement a new federated learning framework. We measure the verification costs under different federated learning rounds on the IRIS dataset. Despite the impact of homomorphic encryption on computational accuracy, the framework proposed in this paper achieves a satisfactory 90% model accuracy. Our framework is highly secure and is expected to further improve the overall efficiency as cryptographic tools continue to evolve.

Accelerated Distributed Approximate Newton Method.

Distributed second-order optimization, as an effective strategy for training large-scale machine learning systems, has been widely investigated due to its low communication complexity. However, the existing distributed second-order optimization algorithms, including distributed approximate Newton (DANE), accelerated inexact DANE (AIDE), and statistically preconditioned accelerated gradient (SPAG), are all required to precisely solve an expensive subproblem up to the target precision. Therefore, this causes these algorithms to suffer from high computation costs and this hinders their development. In this article, we design a novel distributed second-order algorithm called the accelerated distributed approximate Newton (ADAN) method to overcome the high computation costs of the existing ones. Compared with DANE, AIDE, and SPAG, which are constructed based on the relative smooth theory, ADAN's theoretical foundation is built upon the inexact Newton theory. The different theoretical foundations lead to handle the expensive subproblem efficiently, and steps required to solve the subproblem are independent of the target precision. At the same time, ADAN resorts to the acceleration and can effectively exploit the objective function's curvature information, making ADAN to achieve a low communication complexity. Thus, ADAN can achieve both the communication and computation efficiencies, while DANE, AIDE, and SPAG can achieve only the communication efficiency. Our empirical study also validates the advantages of ADAN over extant distributed second-order algorithms.

A Study of Factors Influencing the Adoption of Artificial Intelligence in Crisis Management

This paper presents a study on the Factors Influencing the Adoption of Artificial Intelligence (AI) in Crisis Management. The research identifies 28 AI usage factors categorized into seven groups: Large-Scale Machine Learning, Deep Learning, Reinforcement Learning, Robotics, Computer Vision, Natural Language Processing, and Internet of Things. The study conducted a questionnaire survey among 281 employees at the UAE National Crisis and Emergency Management Authority, using purposive sampling to assess theiropinions regarding the impact of these usage factors on the adoption of AI in crisis management. The collected data underwent descriptive analysis to determine the ranking of AI usage factors within each of the seven groups. In terms of group rankings, Robotic emerged as the top-ranking factor, followed by Reinforcement Learning. Large-Scale Machine Learning occupied the next position, succeeded by Natural Language Processing, Deep Learning, Internet of Things, and Computer Vision, which held the lowest rank. Furthermore, when examining the correlation between these usage factor groups, it was discovered that most of them exhibited strong positive correlations, with correlation coefficients ranging from 0.634 to 0.934. This indicates that changes in one variable are associated with predictable changes in another variable. While this information can be instrumental in understanding relationships and making predictions, it does not establish a causal relationship.