Large-scale Training Research Articles

Evolution and Development of Artificial Intelligence Interpretation Technology in the Age of Large-scale Language Models

With the continuous development of artificial intelligence technology, large-scale language models have become an important breakthrough in the field of interpreting intelligence technology, and the interpreting profession is facing unprecedented technological changes; Through reviewing the relevant research background, it was found that the rise of large models has brought new opportunities and challenges to the development of artificial intelligence interpreting technology; Firstly, the language generation ability of the large model is powerful, which can achieve more accurate and natural interpretation and translation results. Secondly, large models have strong learning abilities and can continuously optimize the performance of interpretation systems through large-scale data training. In addition, the large model also has the characteristic of fast response, which can provide efficient interpretation services in real-time scenarios; The age of large models provides new opportunities and prospects for the transformation and development of artificial intelligence interpretation technology. In the future, with the continuous evolution of large model technology and the expansion of application scenarios, artificial intelligence interpretation technology is expected to play a more important role in cross language communication, international cooperation, and other aspects. However, attention should also be paid to the application limitations of large models and issues such as collaboration with human translators.

Learning explanatory logical rules in non-linear domains: a neuro-symbolic approach

Deep neural networks, despite their capabilities, are constrained by the need for large-scale training data, and often fall short in generalisation and interpretability. Inductive logic programming (ILP) presents an intriguing solution with its data-efficient learning of first-order logic rules. However, ILP grapples with challenges, notably the handling of non-linearity in continuous domains. With the ascent of neuro-symbolic ILP, there’s a drive to mitigate these challenges, synergising deep learning with relational ILP models to enhance interpretability and create logical decision boundaries. In this research, we introduce a neuro-symbolic ILP framework, grounded on differentiable Neural Logic networks, tailored for non-linear rule extraction in mixed discrete-continuous spaces. Our methodology consists of a neuro-symbolic approach, emphasising the extraction of non-linear functions from mixed domain data. Our preliminary findings showcase our architecture’s capability to identify non-linear functions from continuous data, offering a new perspective in neural-symbolic research and underlining the adaptability of ILP-based frameworks for regression challenges in continuous scenarios.

State of the Science: The Unified Protocol for Transdiagnostic Treatment of Emotional Disorders

Before the development of the Unified Protocol for the Transdiagnostic Treatment of Emotional Disorders (UP), evidence-based treatment options for commonly co-occurring anxiety, mood, and related disorders consisted of numerous single-disorder protocols that shared many similarities, reflecting the overlap among these disorders themselves. The UP distilled common elements of cognitive behavioral protocols into a unified intervention directly targeting core temperamental features underlying these disorders of emotion, namely neuroticism and associated emotion dysregulation. The UP has since become a leading “transdiagnostic” treatment for emotional disorders, which is now available in several formats (e.g., individual, group, digital) and has accumulated a strong evidence base, leading to international implementation. There is now also research evidence that the UP can be flexibly applied to a range of clinical presentations, including borderline personality disorder, trauma- and stressor-related disorders, eating disorders, alcohol use disorder, and comorbid chronic physical health conditions. Yet additional research is needed to evaluate the UP in routine clinical settings, with more heterogeneous patient populations, and under circumstances that mirror actual clinical practice. Thus, we must also continue to explore the benefits of large-scale UP training initiatives and implementation in major healthcare systems.

NPR: Nocturnal Place Recognition Using Nighttime Translation in Large-Scale Training Procedures

NPR: Nocturnal Place Recognition Using Nighttime Translation in Large-Scale Training Procedures

Genetic Programming-Based Evolutionary Deep Learning for Data-Efficient Image Classification

Data-efficient image classification is a challenging task that aims to solve image classification using small training data. Neural network-based deep learning methods are effective for image classification, but they typically require large-scale training data and have major limitations such as requiring expertise to design network architectures and having poor interpretability. Evolutionary deep learning is a recent hot topic that combines evolutionary computation with deep learning. However, most evolutionary deep learning methods focus on evolving architectures of neural networks, which still suffer from limitations such as poor interpretability. To address this, this paper proposes a new genetic programming-based evolutionary deep learning approach to data-efficient image classification. The new approach can automatically evolve variable-length models using many important operators from both image and classification domains. It can learn different types of image features from colour or gray-scale images, and construct effective and diverse ensembles for image classification. A flexible multi-layer representation enables the new approach to automatically construct shallow or deep models/trees for different tasks and perform effective transformations on the input data via multiple internal nodes. The new approach is applied to solve five image classification tasks with different training set sizes. The results show that it achieves better performance in most cases than deep learning methods for data-efficient image classification. A deep analysis shows that the new approach has good convergence and evolves models with high interpretability, different lengths/sizes/shapes, and good transferability.

PTMQ: Post-training Multi-Bit Quantization of Neural Networks

The ability of model quantization with arbitrary bit-width to dynamically meet diverse bit-width requirements during runtime has attracted significant attention. Recent research has focused on optimizing large-scale training methods to achieve robust bit-width adaptation, which is a time-consuming process requiring hundreds of GPU hours. Furthermore, converting bit-widths requires recalculating statistical parameters of the norm layers, thereby impeding real-time switching of the bit-width. To overcome these challenges, we propose an efficient Post-Training Multi-bit Quantization (PTMQ) scheme that requires only a small amount of calibration data to perform block-wise reconstruction of multi-bit quantization errors. It eliminates the influence of statistical parameters by fusing norm layers, and supports real-time switching bit-widths in uniform quantization and mixed-precision quantization. To improve quantization accuracy and robustness, we propose a Multi-bit Feature Mixer technique (MFM) for fusing features of different bit-widths to enhance robustness across varying bit-widths. Moreover, we introduced the Group-wise Distillation Loss (GD-Loss) to enhance the correlation between different bit-width groups and further improve the overall performance of PTMQ. Extensive experiments demonstrate that PTMQ achieves comparable performance to existing state-of-the-art post-training quantization methods, while optimizing it speeds up by 100$\times$ compared to recent multi-bit quantization works. Code can be available at https://github.com/xuke225/PTMQ.

Combining Multiple Supervision for Robust Zero-Shot Dense Retrieval

Recently, dense retrieval (DR) models, which represent queries and documents with fixed-width vectors and retrieve relevant ones via nearest neighbor search, have drawn increasing attention from the IR community. However, previous studies have shown that the effectiveness of DR critically relies on sufficient training signals, which leads to severe performance degradation when applied in out-of-domain scenarios, where large-scale training data are usually unavailable. To solve this problem, existing studies adopt a data-augmentation-plus-joint-training paradigm to construct weak/pseudo supervisions on the target domain and combine them with the large-scale human annotated data on the source domain to train the DR models. However, they don't explicitly distinguish the data and the supervision signals in the training process and simply assume that the DR models are mighty enough to capture and memorize different domain knowledge and relevance matching patterns without guidance, which, as shown in this paper, is not true. Based on this observation, we propose a Robust Multi-Supervision Combining strategy (RMSC) that decouples the domain and supervision signals by explicitly telling the DR models how the domain data and supervision signals are combined in the training data with specially designed soft tokens. With the extra soft tokens to store the domain-specific and supervision-specific knowledge, RMSC allows the DR models to conduct retrieval based on human-like relevance matching patterns and target-specific language distribution on the target domain without human annotations. Extensive experiments on zero-shot DR benchmarks show that RMSC significantly improves the ranking performance on the target domain compared to strong DR baselines and domain adaptation methods, while being stable during training and can be combined with query generation or second-stage pre-training.

Abstract 4800: Reducing health disparities for prostate adenocarcinoma by integrating multi-omics data via a multi-modal transfer learning approach

Abstract Prostate adenocarcinoma (PRAD) is the most common subtype of prostate cancer, which is the second leading cause of cancer death in American men. Although significant progress has been made to improve PRAD prognosis, existing studies indicated that Black American men have disproportionately high incidence and mortality rate in prostate cancer compared with non-Hispanic White American men. With artificial intelligence (AI) and machine learning (ML) being increasingly applied to PRAD research and clinical decision-making, PRAD data disparities would introduce bias to AI/ML models and further enhance negative impacts on healthcare towards underrepresented groups. Transfer learning has shown potential to reduce racial disparities in PRAD. However, its performance may be deteriorated due to: (1) its model requires large-scale training samples which are difficult to obtain in clinical settings, and (2) it only uses single-omics data without integrating multi-omics information. To address these concerns, we propose to develop a multi-modal transfer learning model to integrate multi-omics data for reducing health disparities. Specifically, we first investigated two multi-modal ensemble methods, Pearson Correlation Coefficient (PCC) based patient-pairwise similarity, and variational autoencoder (VAE) to integrate different types of omics data. Then, we leveraged a transfer learning model based on domain adaptation to pre-train the model on the majority group (White Americans) and fine-tune the model using the minority group (Black Americans). To further address the imbalanced data among ethnic groups, we explored implementing a data augmentation method, Synthetic Minority Oversampling Technique (SMOTE), to increase the size of minority group data. We evaluated our model on multi-omics data (mRNA, miRNA and methylation) of PRAD from The Cancer Genome Atlas (TCGA) database. Results suggested that our proposed approach achieved better performance of reducing health disparities for Black Americans compared with the mixture model and the independent model as well as the conventional transfer learning model in predicting progression-free interval (PFI) prognosis for PRAD patients. Furthermore, we also demonstrated that SMOTE alleviated the data imbalanced problem and improved the performance of the prognosis classification for ethnic minority groups. In summary, our results demonstrated that our proposed multi-modal transfer learning approach could effectively reduce the performance gap between majority and minority groups in PRAD prognosis, thereby mitigating health disparities. We expect that our proposed multi-model transfer learning framework can be customized and extensible to reduce health disparities in other types of cancer. Citation Format: Lusheng Li, Jieqiong Wang, Shibiao Wan. Reducing health disparities for prostate adenocarcinoma by integrating multi-omics data via a multi-modal transfer learning approach [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 4800.

Adapting low-dose CT denoisers for texture preservation using zero-shot local noise-level matching.

On enhancing the image quality of low-dose computed tomography (LDCT), various denoising methods have achieved meaningful improvements. However, they commonly produce over-smoothed results; the denoised images tend to be more blurred than the normal-dose targets (NDCTs). Furthermore, many recent denoising methods employ deep learning(DL)-based models, which require a vast amount of CT images (or image pairs). Our goal is to address the problem of over-smoothed results and design an algorithm that works regardless of the need for a large amount of training dataset to achieve plausible denoising results. Over-smoothed images negatively affect the diagnosis and treatment since radiologists had developed clinical experiences with NDCT. Besides, a large-scale training dataset is often not available in clinical situations. To overcome these limitations, we propose locally-adaptive noise-level matching (LANCH), emphasizing the output should retain the same noise-level and characteristics to that of the NDCT without additionaltraining. We represent the NDCT image as the pixel-wisely weighted sum of an over-smoothed output from off-the-shelf denoiser (OSD) and the difference between the LDCT image and the OSD output. Herein, LANCH determines a 2D ratio map (i.e., pixel-wise weight matrix) by locally matching the noise-level of output and NDCT, where the LDCT-to-NDCT device flux (mAs) ratio reveals the NDCT noise-level. Thereby, LANCH can preserve important details in LDCT, and enhance the sharpness of the noise-free regions. Note that LANCH can enhance any LDCT denoisers without additional training data (i.e., zero-shot). The proposed method is applicable to any OSD denoisers, reporting significant texture plausibility development over the baseline denoisers in quantitative and qualitative manners. It is surprising that the denoising accuracy achieved by our method with zero-shot denoiser was comparable or superior to that of the best training-based denoisers; our result showed 1% and 33% gains in terms of SSIM and DISTS, respectively. Reader study with experienced radiologists shows significant image quality improvements, a gain of +1.18 on a five-point mean opinion scorescale. In this paper, we propose a technique to enhance any low-dose CT denoiser by leveraging the fundamental physical relationship between the x-ray flux and noise variance. Our method is capable of operating in a zero-shot condition, which means that only a single low-dose CT image is required for the enhancement process. We demonstrate that our approach is comparable or even superior to supervised DL-based denoisers that are trained using numerous CT images. Extensive experiments illustrate that our method consistently improves the performance of all tested LDCT denoisers.

AL-MRIS: An Active Learning-Based Multipath Residual Involution Siamese Network for Few-Shot Hyperspectral Image Classification

In hyperspectral image (HSI) classification scenarios, deep learning-based methods have achieved excellent classification performance, but often rely on large-scale training datasets to ensure accuracy. However, in practical applications, the acquisition of hyperspectral labeled samples is time consuming, labor intensive and costly, which leads to a scarcity of obtained labeled samples. Suffering from insufficient training samples, few-shot sample conditions limit model training and ultimately affect HSI classification performance. To solve the above issues, an active learning (AL)-based multipath residual involution Siamese network for few-shot HSI classification (AL-MRIS) is proposed. First, an AL-based Siamese network framework is constructed. The Siamese network, which has relatively low demand for sample data, is adopted for classification, and the AL strategy is integrated to select more representative samples to improve the model’s discriminative ability and reduce the costs of labeling samples in practice. Then, the multipath residual involution (MRIN) module is designed for the Siamese subnetwork to obtain the comprehensive features of the HSI. The involution operation was used to capture the fine-grained features and effectively aggregate the contextual semantic information of the HSI through dynamic weights. The MRIN module comprehensively considers the local features, dynamic features and global features through multipath residual connections, which improves the representation ability of HSIs. Moreover, a cosine distance-based contrastive loss is proposed for the Siamese network. By utilizing the directional similarity of high-dimensional HSI data, the discriminability of the Siamese classification network is improved. A large number of experimental results show that the proposed AL-MRIS method can achieve excellent classification performance with few-shot training samples, and compared with several state-of-the-art classification methods, the AL-MRIS method obtains the highest classification accuracy.

Self-reported strategy use in working memory tasks

Mnemonic strategies can facilitate working memory performance, but our knowledge on strategy use as a function of task characteristics remains limited. We examined self-reported strategy use in several working memory tasks with pretest data from two large-scale online training experiments. A three-level measure of strategy sophistication (no strategy, maintenance, manipulation) was coded based on participants’ open-ended strategy reports. A considerable portion of participants reported some memory strategy, and strategy sophistication was associated with objective task performance. We found a consistent effect of stimulus type: verbal stimuli (letters or digits) elicited higher strategy sophistication than nonverbal ones (colours or spatial positions). In contrast, the association between task paradigm and strategy sophistication was less consistent in the two experiments. The present results highlight the importance of self-generated strategies in understanding individual differences in working memory performance and the role of stimulus characteristics as one of the task-related determinants of strategy use.

An empirical study on how emotion affects the probability of replies based BERT

The rapid development of the internet, social media, and online forums have become crucial platforms for people to express their views and emotions. Comments are not only a way for users to express their opinions but also play a vital role in promoting discussions and interactions between users, significantly influencing public opinion. This paper aims to explore the impact of emotions on the likelihood of comments receiving replies, deepening the understanding of the role of emotional factors in interactions on social media and online forums. Through large-scale model training and pipeline parallel computation, this paper employs the Bidirectional Encoder Representations from Transformers (BERT) model for learning and prediction, enhancing accuracy and efficiency. The experimental results show that the response rate of negative emotional comments is about 27%, while the response rate of positive emotional comments is about 18%. It means that the comments with negative emotions are more likely to receive replies than those with positive emotions.

A study of data heterogeneity in federated learning

Data-driven artificial intelligence algorithms cannot do without large amounts of training data. However, the characteristics of data privacy and decentralization make constructing large-scale training data costly, which restricts the further application of artificial intelligence algorithms in different downstream fields. To address the above problems, federated learning has gradually attracted more and more research interests in recent years, which aims to utilize decentralized private data for model training while preserving privacy. However, the non-independent and homogeneous distribution of data across devices causes federated learning to face problems such as data imbalance and label bias, which in turn affects the generalization performance of the model. The problem of data heterogeneity has become a major key challenge in federated learning, and this paper aims to explore the impact of data heterogeneity on federated learning and to synthesize recent research results in this area. By analyzing different solution approaches from the aspects of adaptive data distribution, adding regularization terms, contrastive Learning, and multi-task learning, a comprehensive overview is provided for researchers. This paper further summarizes the existing challenges of data heterogeneity in the research field of federated learning and discusses its potential development directions.

Design and Application of Legal Information Systems Based on Big Data Technology

China's exploration of the legal application of big data is still far from thorough enough. This article proposes a universal suitable for hierarchical big data storage systems, which can quickly add new cache policies when needed and provide cache scheduling strategies. The neural training algorithm used in legal information systems implements a complete parallel computing framework for large-scale neural network training, supporting distributed storage and management of large-scale sample data. The experimental results show that the framework has good scalability and fault tolerance, and can quickly train legal information systems, improving their efficiency and response speed. This provides new ideas and methods for the design and development of legal information systems.

Thorough Characterization and Analysis of Large Transformer Model Training At-Scale

Large transformer models have recently achieved great success across various domains. With a growing number of model parameters, a large transformer model training today typically involves model sharding, data parallelism, and model parallelism. Thus, the throughput of large-scale model training depends heavily on the network bandwidth since a combination of model sharding and multiple parallelism strategies incurs various costs. However, prior characterizations of transformer models on high-bandwidth DGX machines that use TFLOPS as a metric may not reflect the performance of a system with lower bandwidth. Furthermore, data and model parallelism reveal significantly distinct training profiles on different system bandwidths at scale and, thus, need a thorough study. In this paper, we provide a bottom-up breakdown of training throughput into compute and communication time, and quantitatively analyze their respective influences on overall end-to-end training scaling. Our evaluation involves an in-depth exploration of data parallelism, scaling up to 512 GPUs with limited bandwidth, and examines three model sharding strategies among six model sizes. We also evaluate three combinations of model parallelism on both high and low bandwidth supercomputing systems. Overall, our work provides a broader perspective on large-scale transformer model training, and our analysis and evaluation yield practical insights for predicting training scaling, shaping the future development of supercomputing system design.

Risk factors for internalizing symptoms: The influence of empathy, theory of mind, and negative thinking processes.

Internalizing symptoms such as elevated stress and sustained negative affect can be important warning signs for developing mental disorders. A recent theoretical framework suggests a complex interplay of empathy, theory of mind (ToM), and negative thinking processes as a crucial risk combination for internalizing symptoms. To disentangle these relationships, this study utilizes neural, behavioral, and self-report data to examine how the interplay between empathy, ToM, and negative thinking processes relates to stress and negative affect. We reanalyzed the baseline data of N = 302 healthy participants (57% female, Mage = 40.52, SDage = 9.30) who participated in a large-scale mental training study, the ReSource project. Empathy and ToM were assessed using a validated fMRI paradigm featuring naturalistic video stimuli and via self-report. Additional self-report scales were employed to measure internalizing symptoms (perceived stress, negative affect) and negative thinking processes (rumination and self-blame). Our results revealed linear associations of self-reported ToM and empathic distress with stress and negative affect. Also, both lower and higher, compared to average, activation in the anterior insula during empathic processing and in the middle temporal gyrus during ToM performance was significantly associated with internalizing symptoms. These associations were dependent on rumination and self-blame. Our findings indicate specific risk constellations for internalizing symptoms. Especially people with lower self-reported ToM and higher empathic distress may be at risk for more internalizing symptoms. Quadratic associations of empathy- and ToM-related brain activation with internalizing symptoms depended on negative thinking processes, suggesting differential effects of cognitive and affective functioning on internalizing symptoms. Using a multi-method approach, these findings advance current research by shedding light on which complex risk combinations of cognitive and affective functioning are relevant for internalizing symptoms.

Block-wise primal-dual algorithms for large-scale doubly penalized ANOVA modeling

For multivariate nonparametric regression, doubly penalized ANOVA modeling (DPAM) has recently been proposed, using hierarchical total variations (HTVs) and empirical norms as penalties on the component functions such as main effects and multi-way interactions in a functional ANOVA decomposition of the underlying regression function. The two penalties play complementary roles: the HTV penalty promotes sparsity in the selection of basis functions within each component function, whereas the empirical-norm penalty promotes sparsity in the selection of component functions. To facilitate large-scale training of DPAM using backfitting or block minimization, two suitable primal-dual algorithms are developed, including both batch and stochastic versions, for updating each component function in single-block optimization. Existing applications of primal-dual algorithms are intractable for DPAM with both HTV and empirical-norm penalties. The validity and advantage of the stochastic primal-dual algorithms are demonstrated through extensive numerical experiments, compared with their batch versions and a previous active-set algorithm, in large-scale scenarios.

Parallel programming: Driving the computational surge in AI

With the meteoric ascent and ongoing advancements in the realm of Artificial Intelligence (AI), there's an escalating demand for potent computational capabilities. Meeting the exacting computational demands of burgeoning AI technologies has become a pivotal topic in contemporary research. This study is anchored on the theme: "Impact of Parallel Programming on AI." The objective is to delve into the potentialities and value that parallel programming technology might introduce to the AI sector. To deeply probe the integration of parallel programming in AI, we've employed a meticulous research methodology. This process disentangles and scrutinizes the prevalent parallel programming techniques and their tangible applications within the AI sphere. Such a method offers a nuanced grasp of the fusion of parallel programming with AI and the distinct advantages that ensue. Findings reveal a substantial enhancement in the efficacy of AI models when they leverage parallel programming techniques. This is especially salient in scenarios involving large-scale data training and intricate model architectures. Most notably, parallel programming dramatically slashes AI training durations. This swift training paves the way for rapid iterations and fine-tuning. In conclusion, parallel programming emerges as a game-changer in the AI domain. It doesn't merely amplify AI performance but also lays a robust technical bedrock for AI's sustained and swift evolution. This opens up a vista of novel prospects and avenues for forthcoming exploration and application in AI.

PV output estimation method of power distribution station area based on federated learning framework and improved transformer neural network

In contemporary distribution networks (DNs), characterized by extensive integration of distributed energy, the photovoltaic (PV) power output data from the distribution station areas become crucial for system planning and operational optimization. Since many PVs are installed behind-the-meter (BTM), it is difficult to directly obtain PV power data through measurement devices. Therefore, it is important to estimate the BTM PV power from the aggregating data that can be directly obtained. However, the existing estimation methods usually require centralized large-scale data training, which brings certain privacy leakage risks. In order to solve these problems, we propose a federated learning-based improved Transformer Neural Network strategy to estimate BTM PV generation at the community level with data privacy protection. Initially, enhanced Transformer neural networks, employing a fused-attention mechanism, are deployed to precisely delineate the solar power generation pattern. Subsequently, federated learning principles facilitate the sharing of specific parameters among multiple edge endpoints and a central server. This model bifurcates into two layers: an individual layer, where parameters are retained locally, and an exchange layer, where parameters are collectively shared and conveyed through momentum aggregation. This dual-layer structure effectively synchronizes the capture of both unique and common characteristics. The test on the Australian residential load dataset verifies the effectiveness of the proposed method.

Finding Your Voice: A Large-Scale Nursing Training in Speaking up and Listening Skills.

Training in speaking up skills for nursing professionals provides the groundwork for promoting a culture of curiosity and inquiry. At the foundation of speaking up is psychological safety for team members to disagree, offer alternative ideas, and ask questions. A large-scale training session, Finding Your Voice: Speaking Up & Listening Skills, was designed and delivered to 1,306 nurses at an academic medical center. Most participants reported that the session met the training goals (97%). There was an increase in participants' confidence in speaking up (pretraining 69% ± 19%; posttraining 73% ± 15%; p < .0001), and listening (pretraining 68% ± 19%; post-training 74% ± 15%; p < .0001). Leaders showed greater increases in confidence in listening and responding to nurses speaking up (leaders 5.6% vs. nonleaders 4.2%, p < .00001). Nurses benefited from an interactive educational experience by practicing strategies for speaking up and listening. [J Contin Educ Nurs. 2024;55(6):309-316.].