Mixture Of Experts Research Articles

HDformer: A Higher-Dimensional Transformer for Detecting Diabetes Utilizing Long-Range Vascular Signals

Diabetes mellitus is a global concern, and early detection can prevent serious complications. 50% of those with diabetes live undiagnosed, disproportionately afflicting low-income groups. Non-invasive methods have emerged for timely detection; however, their limited accuracy constrains clinical usage. In this research, we present a novel Higher Dimensional Transformer (HDformer), the first Transformer-based architecture which utilizes long-range photoplethysmography (PPG) to detect diabetes. The long-range PPG maximizes signal contextual information when compared to the less-than 30 second signals commonly used in existing research. To increase the computational efficiency of HDformer’s long-range processing, a new attention module, Time Square Attention (TSA), is invented to achieve linear computational complexity with respect to the token volume while retaining the local/global dependencies. TSA converts the 1D inputs into 2D representations, grouping the adjacent points into a single 2D token. It then generates dynamic patches and feeds them into a gated mixture-of-experts (MoE) network, optimizing the learning on different attention areas. HDformer achieves state-of-the-art results (sensitivity 98.4, accuracy 97.3, specificity 92.8, AUC 0.929) on the standard MIMIC-III dataset, surpassing existing research. Furthermore, we develop an end-to-end solution where a low-cost wearable is prototyped to connect with the HDformer in the Cloud via a mobile app. This scalable, convenient, and affordable approach provides instantaneous detection and continuous monitoring for individuals. It aids doctors in easily screening for diabetes and safeguards underprivileged communities. The enhanced versatility of HDformer allows for efficient processing and learning of long-range signals in general one-dimensional time-series sequences, particularly for all biomedical waveforms.

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

Building scalable vision-language models to learn from diverse, multimodal data remains an open challenge. In this paper, we introduce an Efficient Vision-languagE foundation model, namely EVE, which is one unified multimodal Transformer pre-trained solely by one unified pre-training task. Specifically, EVE encodes both vision and language within a shared Transformer network integrated with modality-aware sparse Mixture-of-Experts (MoE) modules, which capture modality-specific information by selectively switching to different experts. To unify pre-training tasks of vision and language, EVE performs masked signal modeling on image-text pairs to reconstruct masked signals, i.e., image pixels and text tokens, given visible signals. This simple yet effective pre-training objective accelerates training by 4x compared to the model pre-trained with Image-Text Contrastive and Image-Text Matching losses. Owing to the combination of the unified architecture and pre-training task, EVE is easy to scale up, enabling better downstream performance with fewer resources and faster training speed. Despite its simplicity, EVE achieves state-of-the-art performance on various vision-language downstream tasks, including visual question answering, visual reasoning, and image-text retrieval.

A Model for Customer Opinion Mining and Sentiment Classification using a Mixture of Experts Machine Learning Model

This paper presents customer opinion mining technique based on a Mixture of Experts (MoE) machine learning model. The approach allows a corpus from open source data repositories to be classified into positive, negative and neutral sentiments as the case may be and in a predictive manner. The results of simulations showed that the proposed MoE approach can effectively be used as a core tool in opinion mining and also serve in decision making by appropriate categorizations. In particular, it was found that the use of higher epoch sizes greatly enhances the performance of the MoE by reducing perplexity and error cost margins to appreciable levels. Thus, the MoE presents a promising candidate for customer opinion mining particularly in business product development environments.

Functional mixtures-of-experts

AbstractWe consider the statistical analysis of heterogeneous data for prediction, in situations where the observations include functions, typically time series. We extend the modeling with mixtures-of-experts (ME), as a framework of choice in modeling heterogeneity in data for prediction with vectorial observations, to this functional data analysis context. We first present a new family of ME models, named functional ME (FME), in which the predictors are potentially noisy observations, from entire functions. Furthermore, the data generating process of the predictor and the real response, is governed by a hidden discrete variable representing an unknown partition. Second, by imposing sparsity on derivatives of the underlying functional parameters via Lasso-like regularizations, we provide sparse and interpretable functional representations of the FME models called iFME. We develop dedicated expectation–maximization algorithms for Lasso-like regularized maximum-likelihood parameter estimation strategies to fit the models. The proposed models and algorithms are studied in simulated scenarios and in applications to two real data sets, and the obtained results demonstrate their performance in accurately capturing complex nonlinear relationships and in clustering the heterogeneous regression data.

Switching Self-Attention Text Classification Model with Innovative Reverse Positional Encoding for Right-to-Left Languages: A Focus on Arabic Dialects

Transformer models have emerged as frontrunners in the field of natural language processing, primarily due to their adept use of self-attention mechanisms to grasp the semantic linkages between words in sequences. Despite their strengths, these models often face challenges in single-task learning scenarios, particularly when it comes to delivering top-notch performance and crafting strong latent feature representations. This challenge is more pronounced in the context of smaller datasets and is particularly acute for under-resourced languages such as Arabic. In light of these challenges, this study introduces a novel methodology for text classification of Arabic texts. This method harnesses the newly developed Reverse Positional Encoding (RPE) technique. It adopts an inductive-transfer learning (ITL) framework combined with a switching self-attention shared encoder, thereby increasing the model’s adaptability and improving its sentence representation accuracy. The integration of Mixture of Experts (MoE) and RPE techniques empowers the model to process longer sequences more effectively. This enhancement is notably beneficial for Arabic text classification, adeptly supporting both the intricate five-point and the simpler ternary classification tasks. The empirical evidence points to its outstanding performance, achieving accuracy rates of 87.20% for the HARD dataset, 72.17% for the BRAD dataset, and 86.89% for the LABR dataset, as evidenced by the assessments conducted on these datasets.

Generalizable Segmentation of COVID-19 Infection From Multi-Site Tomography Scans: A Federated Learning Framework

COVID-19-like pandemics are a major threat to the global health system that causes a lot of deaths across ages. Large-scale medical images (i.e., X-rays, computed tomography (CT)) dataset is favored to the accuracy of deep learning (DL) in the screening of COVID-19-like pneumonia. The cost, time, and efforts for acquiring and annotating, for instance, large CT datasets make it impossible to obtain large numbers of samples from a single institution. The research attentions have been moved toward sharing medical images from numerous medical institutions. However, owing to the necessity to preserve the privacy of the data of a patient, it is challenging to build a centralized dataset from many institutions, especially during the pandemic. More. The difference in the data acquisition process from one institution to another brings another challenge known as distribution heterogeneity. This paper presents a novel federated learning framework, called Federated Multi-Site COVID-19 (FEDMSCOV), for efficient, generalizable, and privacy-preserved segmentation of COVID-19 infection from multi-site data. In FEDMSCOV, a novel is local drift smoothing (LDS) module encodes the input from feature space to frequency space, aiming to suppress the modules that are not conducive to generalization. Given the smoothed local updated, FEDMSCOV presents a novel Mixture-of-Expert (MoE) scheme to resolve global shift in parameters. An adapted differential privacy method is applied to design and protect the privacy of local updates during the training. Experimental evaluation on a large-scale multi-institutional COVID-19 dataset demonstrated the efficiency of the proposed framework over competing learning approaches with statistical significance.

Emergent communication of multimodal deep generative models based on Metropolis-Hastings naming game.

Deep generative models (DGM) are increasingly employed in emergent communication systems. However, their application in multimodal data contexts is limited. This study proposes a novel model that combines multimodal DGM with the Metropolis-Hastings (MH) naming game, enabling two agents to focus jointly on a shared subject and develop common vocabularies. The model proves that it can handle multimodal data, even in cases of missing modalities. Integrating the MH naming game with multimodal variational autoencoders (VAE) allows agents to form perceptual categories and exchange signs within multimodal contexts. Moreover, fine-tuning the weight ratio to favor a modality that the model could learn and categorize more readily improved communication. Our evaluation of three multimodal approaches - mixture-of-experts (MoE), product-of-experts (PoE), and mixture-of-product-of-experts (MoPoE)-suggests an impact on the creation of latent spaces, the internal representations of agents. Our results from experiments with the MNIST + SVHN and Multimodal165 datasets indicate that combining the Gaussian mixture model (GMM), PoE multimodal VAE, and MH naming game substantially improved information sharing, knowledge formation, and data reconstruction.

Enhancing diversity for logical table‐to‐text generation with mixture of experts

AbstractLogical table‐to‐text generation is a task within the realm of natural language generation (NLG) that aims to generate coherent and logically faithful sentences based on tables. Unlike conventional NLG tasks, this task demands not only surface‐level fluency but also a high degree of logic‐level fidelity in the generated outputs. Current table‐to‐text systems grapple with various quality issues, such as repetitive generation, insufficient reasoning and limited complexity. Therefore, we introduce LogicMoE, a dedicated Mixture‐of‐Experts (MoE) model tailored for logical table‐to‐text generation. The primary objective of LogicMoE is to enrich the diversity of generated sentences from both semantic and logical perspectives. In particular, each expert within the model serves as a specialized generator responsible for generating sentences of a specific logical type. Additionally, we propose and employ novel evaluation metrics to comprehensively assess the diversity of generated outputs. Our experimental results showcase LogicMoE's superiority with absolute improvements of 0.8 and 2.2 in BLEU‐3 over the strong baselines on LogicNLG and Logic2Text datasets, respectively, driving the state‐of‐the‐art performance to a new level. Furthermore, we highlight its inherent advantages in terms of diversity and controllability, signifying its potential to spearhead advancements in logical table‐to‐text generation applications.

Switch-Transformer Sentiment Analysis Model for Arabic Dialects That Utilizes a Mixture of Experts Mechanism

In recent years, models such as the transformer have demonstrated impressive capabilities in the realm of natural language processing. However, these models are known for their complexity and the substantial training they require. Furthermore, the self-attention mechanism within the transformer, designed to capture semantic relationships among words in sequences, faces challenges when dealing with short sequences. This limitation hinders its effectiveness in five-polarity Arabic sentiment analysis (SA) tasks. The switch-transformer model has surfaced as a potential substitute. Nevertheless, when employing one-task learning for their training, these models frequently face challenges in presenting exceptional performances and encounter issues when producing resilient latent feature representations, particularly in the context of small-size datasets. This challenge is particularly prominent in the case of the Arabic dialect, which is recognized as a low-resource language. In response to these constraints, this research introduces a novel method for the sentiment analysis of Arabic text. This approach leverages multi-task learning (MTL) in combination with the switch-transformer shared encoder to enhance model adaptability and refine sentence representations. By integrating a mixture of experts (MoE) technique that breaks down the problem into smaller, more manageable sub-problems, the model becomes skilled in managing extended sequences and intricate input–output relationships, thereby benefiting both five-point and three-polarity Arabic sentiment analysis tasks. The proposed model effectively identifies sentiment in Arabic dialect sentences. The empirical results underscore its exceptional performance, with accuracy rates reaching 84.02% for the HARD dataset, 67.89% for the BRAD dataset, and 83.91% for the LABR dataset, as demonstrated by the evaluations conducted on these datasets.

Overcoming linguistic barriers in code assistants: creating a Qlora adapter to improve support for russian-language code writing instructions

This article describes an approach to learning and evaluating an adapter model for the popular zephyr-7b-beta language model. The adapter was designed to improve the performance of the basic model in tasks related to programming and understanding the Russian language. Taking into account the high quality of the original model's work in problems in English, the purpose of the study was to expand its linguistic and technical spectrum. The proposed adapter was trained using a large and diverse dataset, including question-answer pairs related to programming, as well as a text with a discussion of the code in Russian. The applied training methodology ensures an improvement in the quality of the model's responses in understanding and generating Python code based on Russian-language instructions. The authors evaluated the performance of the basic model with the adapter installed using various metrics, comparing it with the basic model, as well as with other advanced models in this area. The results obtained by the authors showed significant improvement both in tasks related to writing Python code and in processing the Russian language, confirming the effectiveness of the proposed adapter. The analysis of the experiment results showed that the adapter not only adds new functionality to the model but also significantly changes the priority of the model's output in favor of the instructions that were included during the retraining stage. To eliminate the fact of changing the model's output priority mentioned above, the following methods are proposed: 1) Using a different format for writing the query (prompt) when formatting the dataset for training the model, different from the original format in Zephyr. 2) Including instructions from the first stage of fine-tuning (retraining) in the instruction set. 3) Creating a "mixture of experts" from several different models, where each model will specialize in the most accurate solution to one or several tasks from the overall range of solvable tasks. The methods of retraining the model without losing the original characteristics are promising directions for future research in the field of resource-efficient modification of large language models. In the conducted experiment, a large language model "HuggingFaceH4/zephyr-7b-beta" was successfully fine-tuned (using the QLoRA adapter creation method) to improve its capabilities in writing Python code based on Russian instructions and providing explanations in Russian. The testing results showed that when the created adapter is installed, the model's text generation priority shifts towards generating code with explanations. Synthetic tests also demonstrated significant improvements in the model's ability to solve programming and mathematical problems. Therefore, it is advisable to consider the possibility of changing adapters depending on the type of task being solved. Changing the adapter installed on the model (loaded into memory) does not take a significant amount of time, which allows for the creation of a set of adapters tailored to specific tasks and their interchange based on the type of tasks being solved. Changing adapters instead of models eliminates the need to load multiple full versions of the model into the memory of the computational accelerator, which, depending on the implementation method, either significantly reduces the use of memory or the model loading time. Thus, the use of adapters speeds up training and optimizes the utilization of computational resources during model operation. This presented research contributes to the current efforts in the field of machine learning and natural language processing towards creating more versatile AI models. The observed potential of adapter models in improving the quality of responses in specific domains without the need for extensive retraining helps meet the growing demand for models for multilingual natural language processing and code generation. In the conducted experiment, an assessment was made of the possibility of retraining a model that had already been trained on a set of instructions. The methods of additional training of a language model with new instructions without losing the ability to perform original instructions are not well-studied, which is why a decision was made to test the feasibility and effectiveness of such retraining. Based on the description of the LoRA method, this approach is possible because during such training, the original weights of the model remain unchanged and are only supplemented with new coefficients from the adapter model (the total number of weights equals that of the base model), which was trained on new instructions. The experiment determined that multilingual adaptation of language models is viable, and that it also could be implemented using parametrically efficient methods on comparatively low-end hardware.

Toward Scalable Generative AI via Mixture of Experts in Mobile Edge Networks

Toward Scalable Generative AI via Mixture of Experts in Mobile Edge Networks

Spatial-Frequency Adaptive Remote Sensing Image Dehazing With Mixture of Experts

Spatial-Frequency Adaptive Remote Sensing Image Dehazing With Mixture of Experts

High Performance Breast Cancer Diagnosis From Mammograms Using Mixture of Experts With EfficientNet Features (MoEffNet)

High Performance Breast Cancer Diagnosis From Mammograms Using Mixture of Experts With EfficientNet Features (MoEffNet)

Deep learning based prediction of traffic peaks in mobile networks

In mobile networks, it is essential to configure networks more efficiently to provide mobile users with services having better quality. For the adjacent cells, sometimes the mobile traffic concentrates in a single cell, leading to traffic imbalance; in many cases, the detection of imbalance is strongly related to the prediction of traffic peaks, which is extremely difficult because many peaks appear suddenly for no apparent reason. To better predict the peaks and traffic imbalance, we propose two novel mobile traffic predictors. The first is a Mixture of Experts (MoE) model which yields significantly better peak prediction along with excellent interpretability by establishing a cooperation mechanism between different experts, whereas the second predictor is a lightweight Multilayer Perceptron (MLP) which can obtain similar peak forecasting performance but operating more flexibly and consuming less computational power. The obtained predictions are then used to aid the predictive detection of traffic imbalance. To this end, we first perform a large-scale analysis of mobile traffic and then propose different approaches to detect the future traffic imbalance based on the predictions. To evaluate the performance of the proposed approaches, we have conducted extensive experiments on real-world mobile network datasets; all models are implemented in Python 3.8.8 with Pytorch 1.7.0, and they were run on a computer with i7-7700K CPU @ 4.2 GHz running Linux, 64 GB memory and a single Nvidia TITAN X Pascal 12GB GPU. The proposed models are compared with 10 widely used deep learning based predictors, and the results show that our approaches have significantly improved the sensitivity (peak prediction accuracy) from 39.3% to 58.5% (i.e., roughly a 50% increase) along with excellent interpretability. Moreover, the predictions are further used by the predictive detection approach, where the best multi-model approach improves the predictive classification accuracy of congestion (imbalance prediction accuracy) by 10.3% in contrast with the naive approach; this result also verifies the effectiveness of our peak predictions.

Multimodal Gated Mixture of Experts Using Whole Slide Image and Flow Cytometry for Multiple Instance Learning Classification of Lymphoma

In this study, we present a deep-learning-based multimodal classification method for lymphoma diagnosis in digital pathology, which utilizes a whole slide image (WSI) as the primary image data and flow cytometry (FCM) data as auxiliary information. In pathological diagnosis of malignant lymphoma, FCM serves as valuable auxiliary information during the diagnosis process, offering useful insights into predicting the major class (superclass) of subtypes. By incorporating both images and FCM data into the classification process, we can develop a method that mimics the diagnostic process of pathologists, enhancing the explainability. In order to incorporate the hierarchical structure between superclasses and their subclasses, the proposed method utilizes a network structure that effectively combines the mixture of experts (MoE) and multiple instance learning (MIL) techniques, where MIL is widely recognized for its effectiveness in handling WSIs in digital pathology. The MoE network in the proposed method consists of a gating network for superclass classification and multiple expert networks for (sub)class classification, specialized for each superclass. To evaluate the effectiveness of our method, we conducted experiments involving a six-class classification task using 600 lymphoma cases. The proposed method achieved a classification accuracy of 72.3%, surpassing the 69.5% obtained through the straightforward combination of FCM and images, as well as the 70.2% achieved by the method using only images. Moreover, the combination of multiple weights in the MoE and MIL allows for the visualization of specific cellular and tumor regions, resulting in a highly explanatory model that cannot be attained with conventional methods. It is anticipated that by targeting a larger number of classes and increasing the number of expert networks, the proposed method could be effectively applied to the real problem of lymphoma diagnosis.

A MLP-Mixer and mixture of expert model for remaining useful life prediction of lithium-ion batteries

Accurately predicting the Remaining Useful Life (RUL) of lithium-ion batteries is crucial for battery management systems. Deep learning-based methods have been shown to be effective in predicting RUL by leveraging battery capacity time series data. However, the representation learning of features such as long-distance sequence dependencies and mutations in capacity time series still needs to be improved. To address this challenge, this paper proposes a novel deep learning model, the MLP-Mixer and Mixture of Expert (MMMe) model, for RUL prediction. The MMMe model leverages the Gated Recurrent Unit and Multi-Head Attention mechanism to encode the sequential data of battery capacity to capture the temporal features and a re-zero MLP-Mixer model to capture the high-level features. Additionally, we devise an ensemble predictor based on a Mixture-of-Experts (MoE) architecture to generate reliable RUL predictions. The experimental results on public datasets demonstrate that our proposed model significantly outperforms other existing methods, providing more reliable and precise RUL predictions while also accurately tracking the capacity degradation process. Our code and dataset are available at the website of github.

Improving speech emotion recognition by fusing self-supervised learning and spectral features via mixture of experts

Speech Emotion Recognition (SER) is an important area of research in speech processing that aims to identify and classify emotional states conveyed through speech signals. Recent studies have shown considerable performance in SER by exploiting deep contextualized speech representations from self-supervised learning (SSL) models. However, SSL models pre-trained on clean speech data may not perform well on emotional speech data due to the domain shift problem. To address this problem, this paper proposes a novel approach that simultaneously exploits an SSL model and a domain-agnostic spectral feature (SF) through the Mixture of Experts (MoE) technique. The proposed approach achieves the state-of-the-art performance on weighted accuracy compared to other methods in the IEMOCAP dataset. Moreover, this paper demonstrates the existence of the domain shift problem of SSL models in the SER task.

Pitfalls of Using Multinomial Regression Analysis to Identify Class-Structure-Relevant Variables in Biomedical Data Sets: Why a Mixture of Experts (MOE) Approach Is Better

Recent advances in mathematical modeling and artificial intelligence have challenged the use of traditional regression analysis in biomedical research. This study examined artificial data sets and biomedical data sets from cancer research using binomial and multinomial logistic regression. The results were compared with those obtained with machine learning models such as random forest, support vector machine, Bayesian classifiers, k-nearest neighbors, and repeated incremental clipping (RIPPER). The alternative models often outperformed regression in accurately classifying new cases. Logistic regression had a structural problem similar to early single-layer neural networks, which limited its ability to identify variables with high statistical significance for reliable class assignments. Therefore, regression is not per se the best model for class prediction in biomedical data sets. The study emphasizes the importance of validating selected models and suggests that a “mixture of experts” approach may be a more advanced and effective strategy for analyzing biomedical data sets.

Further exploration of the effects of time-varying covariate in growth mixture models with nonlinear trajectories.

Growth mixture modeling (GMM) is an analytical tool for identifying multiple unobserved sub-populations in longitudinal processes. In particular, it describes change patterns within each latent sub-population and investigates between-individual differences in within-individual change for each sub-group. A key research interest in using GMMs is examining how covariates influence the heterogeneity in change patterns. Liu & Perera (2022b) extended mixture-of-experts (MoE) models, which primarily focus on time-invariant covariates, to allow covariates to account for both within-group and between-group differences and investigate the heterogeneity in nonlinear trajectories. The present study further extends Liu & Perera, 2022b by examining the effects of time-varying covariates (TVCs) on trajectory heterogeneity. Specifically, we propose methods to decompose a TVC into an initial trait (the baseline value of the TVC) and a set of temporal states (interval-specific slopes or changes of the TVC). The initial trait is allowed to account for within-group differences in growth factors of trajectories (i.e., baseline effect), while the temporal states are allowed to impact observed values of a longitudinal process (i.e., temporal effects). We evaluate the proposed models using a simulation study and real-world data analysis. The simulation study demonstrates that the proposed models are capable of separating trajectories into several clusters and generally producing unbiased and accurate estimates with target coverage probabilities. The proposed models reveal the heterogeneity in initial trait and temporal states of reading ability across latent classes of students' mathematics performance. Additionally, the baseline and temporal effects on mathematics development of reading ability are also heterogeneous across the clusters of students.

Accelerating Chemical Kinetics Calculations With Physics Informed Neural Networks

Abstract Detailed chemical kinetics calculations can be very computationally expensive, and so various approaches have been used to speed up combustion calculations. Deep neural networks (DNNs) are one promising approach that has seen significant development recently. Standard DNNs, however, do not necessarily follow physical constraints such as conservation of mass. Physics Informed Neural Networks (PINNs) are a class of neural networks that have physical laws embedded within the training process to create networks that follow those physical laws. A new PINN-based DNN approach to chemical kinetics modeling has been developed to make sure mass fraction predictions adhere to the conservation of atomic species. The approach also utilizes a mixture-of-experts (MOE) architecture where the data is distributed on multiple subnetworks followed by a softmax selective layer. The MOE architecture allows the different subnetworks to specialize in different thermochemical regimes, such as early stage ignition reactions or post-flame equilibrium chemistry, then the softmax layer smoothly transitions between the subnetwork predictions. This modeling approach was applied to the prediction of methane-air combustion using the GRI-Mech 3.0 as the reference mechanism. The training database was composed of data from 0D ignition delay simulations under initial conditions of 0.2–50 bar pressure, 500–2000 K temperature, an equivalence ratio between 0 and 2, and an N2-dilution percentage of up to 50%. A wide variety of network sizes and architectures of between 3 and 20 subnetworks and 6,600 to 77,000 neurons were tested. The resulting networks were able to predict 0D combustion simulations with similar accuracy and atomic mass conservation as standard kinetics solvers while having a 10-50× speedup in online evaluation time using CPUs, and on average over 200× when using a GPU.