Mixture Of Experts Research Articles

Leveraging Mixture of Experts and Deep Learning-Based Data Rebalancing to Improve Credit Fraud Detection

Credit card fraud detection is a critical challenge in the financial sector due to the rapidly evolving tactics of fraudsters and the significant class imbalance betweenegitimate and fraudulent transactions. Traditional models, while effective to some extent, often suffer from high false positive rates and fail to generalize well to emerging fraud patterns. In this paper, we propose a novel approach that integrates a Mixture of Experts (MoE) model with a Deep Neural Network-based Synthetic Minority Over-sampling Technique (DNN-SMOTE) to enhance fraud detection performance. The MoE modeleverages multiple specialized expert networks, each trained to detect specific types of fraud, while the DNN-SMOTE generates high-quality synthetic samples to address the class imbalance. Our experimental results on a publicly available dataset demonstrate that the proposed method achieves a classification accuracy of 99.93%, a true positive rate of 84.69%, and a true negative rate of 99.95%. The Matthews Correlation Coefficient (MCC) of 0.7883 further highlights the model’s balanced performance in detecting fraudulent transactions. These results underscore the effectiveness of combining MoE with DNN-SMOTE, offering a robust solution for real-world credit card fraud detection scenarios.

Enhancing Superconductor Critical Temperature Prediction: A Novel Machine Learning Approach Integrating Dopant Recognition.

Doping plays a crucial role in determining the critical temperature (Tc) of superconductors, yet accurately predicting its effects remains a significant challenge. Here, we introduce a novel doping descriptor that captures the complex influence of dopants on superconductivity. By integrating the doping descriptor with elemental and physical features within a Mixture of Experts (MoE) model, we achieve a remarkable R2 of 0.962 for Tc prediction, surpassing all published prediction models. Our approach successfully identifies optimal doping levels in the Bi2-xPbxSr2Ca2-yCuyOz system, with predictions closely aligning with experimental results. Leveraging this model, we screen compounds from the Inorganic Crystal Structure Database and employ a generative approach to explore new doped superconductors. This process reveals 40 promising candidates for high Tc superconductivity among existing and hypothetical doped materials. By explicitly accounting for doping effects, our method offers a powerful tool for guiding the experimental discovery of new superconductors, potentially accelerating progress in high-temperature superconductivity research and opening new avenues for material design.

Category-guided multi-interest collaborative metric learning with representation uniformity constraints

Multi-interest collaborative metric learning has recently emerged as an effective approach to modeling the multifaceted interests of a user in recommender systems. However, two issues remain unexplored. (1) There is no explicit guidance for the matching of an item against multiple interest vectors of a user. (2) The desired property of item representations with respect to their categories is overlooked, resulting in that different categories of items are mixed up in the latent space. To overcome these issues, we devise a Category-guided Multi-interest Collaborative Metric Learning model (CMCML) with representation uniformity constraints. CMCML is designed as a novel category-guided Mixture-of-Experts (MoE) architecture, where the gating network leverages the item category to guide the matching of an item against multiple interest vectors of a user, encouraging items with the same category to approach the same interest vector. In addition, we design a user multi-interest uniformity loss and a category-aware item uniformity loss: The former aims to avoid representation degeneration by enlarging the difference among multiple interest vectors of the same user; the latter is tailored to push different categories of items apart in the latent space. Quantitative experiments on Ciao, Epinions and TaFeng demonstrate that our CMCML improves the value of NDCG@20 by 12.41%, 10.89% and 10.39% respectively, compared to other state-of-the-art collaborative metric learning methods. Further qualitative analyses reveal that our CMCML yields a better representation space where items from distinct categories are arranged in different regions with high density.

Interpretable Mixture of Experts for Decomposition Network on Server Performance Metrics Forecasting

The accurate forecasting of server performance metrics, such as CPU utilization, memory usage, and network bandwidth, is critical for optimizing resource allocation and ensuring system reliability in large-scale computing environments. In this paper, we introduce the Mixture of Experts for Decomposition Kolmogorov–Arnold Network (MOE-KAN), a novel approach designed to improve both the accuracy and interpretability of server performance prediction. The MOE-KAN framework employs a decomposition strategy that breaks down complex, nonlinear server performance patterns into simpler, more interpretable components, facilitating a clearer understanding of how predictions are made. By leveraging a Mixture of Experts (MOE) model, trend and residual components are learned by specialized experts, whose outputs are transparently combined to form the final prediction. The Kolmogorov–Arnold Network further enhances the model’s ability to capture intricate input–output relationships while maintaining transparency in its decision-making process. Experimental results on real-world server performance datasets demonstrate that MOE-KAN not only outperforms traditional models in terms of accuracy but also provides a more trustworthy and interpretable forecasting framework. This makes it particularly suitable for real-time server management and capacity planning, offering both reliability and interpretability in predictive models.

Dynamic Determination Method of Line Drop Compensator Parameters for Voltage Regulators Based on Mixture of Experts Using Real‐Time Information

In this paper, a method for determining line drop compensator (LDC) parameters for step voltage regulators and on‐load tap changers is proposed to avoid voltage violations in distribution systems with voltage fluctuations due to photovoltaic (PV) generation and electric vehicle (EV) charging. Distribution system operators need to effectively solve voltage problems caused by the widespread installation of distributed energy resources with the existing voltage regulators to construct an efficient and rational infrastructure. Our focus was on improving the method for determining the LDC parameters for the existing voltage regulators based on LDC control. A mechanism to dynamically change the LDC parameters depending on the situations in the distribution system was developed by using a mixture of experts, one of the machine learning techniques, and real‐time voltage and power flow information from information technology switches. The power flow calculations were performed using a distribution system model constructed based on the topology, loads, and generation characteristics of an actual distribution system in the Hokuriku region. The effectiveness of the proposed method was evaluated in terms of its improved effect on PV and EV hosting capacity, as well as the impact on the frequency of tap operations of voltage regulators. © 2024 The Author(s). IEEJ Transactions on Electrical and Electronic Engineering published by Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

Equipping computational pathology systems with artifact processing pipelines: a showcase for computation and performance trade-offs

BackgroundHistopathology is a gold standard for cancer diagnosis. It involves extracting tissue specimens from suspicious areas to prepare a glass slide for a microscopic examination. However, histological tissue processing procedures result in the introduction of artifacts, which are ultimately transferred to the digitized version of glass slides, known as whole slide images (WSIs). Artifacts are diagnostically irrelevant areas and may result in wrong predictions from deep learning (DL) algorithms. Therefore, detecting and excluding artifacts in the computational pathology (CPATH) system is essential for reliable automated diagnosis.MethodsIn this paper, we propose a mixture of experts (MoE) scheme for detecting five notable artifacts, including damaged tissue, blur, folded tissue, air bubbles, and histologically irrelevant blood from WSIs. First, we train independent binary DL models as experts to capture particular artifact morphology. Then, we ensemble their predictions using a fusion mechanism. We apply probabilistic thresholding over the final probability distribution to improve the sensitivity of the MoE. We developed four DL pipelines to evaluate computational and performance trade-offs. These include two MoEs and two multiclass models of state-of-the-art deep convolutional neural networks (DCNNs) and vision transformers (ViTs). These DL pipelines are quantitatively and qualitatively evaluated on external and out-of-distribution (OoD) data to assess generalizability and robustness for artifact detection application.ResultsWe extensively evaluated the proposed MoE and multiclass models. DCNNs-based MoE and ViTs-based MoE schemes outperformed simpler multiclass models and were tested on datasets from different hospitals and cancer types, where MoE using (MobileNet) DCNNs yielded the best results. The proposed MoE yields 86.15 % F1 and 97.93% sensitivity scores on unseen data, retaining less computational cost for inference than MoE using ViTs. This best performance of MoEs comes with relatively higher computational trade-offs than multiclass models. Furthermore, we apply post-processing to create an artifact segmentation mask, a potential artifact-free RoI map, a quality report, and an artifact-refined WSI for further computational analysis. During the qualitative evaluation, field experts assessed the predictive performance of MoEs over OoD WSIs. They rated artifact detection and artifact-free area preservation, where the highest agreement translated to a Cohen Kappa of 0.82, indicating substantial agreement for the overall diagnostic usability of the DCNN-based MoE scheme.ConclusionsThe proposed artifact detection pipeline will not only ensure reliable CPATH predictions but may also provide quality control. In this work, the best-performing pipeline for artifact detection is MoE with DCNNs. Our detailed experiments show that there is always a trade-off between performance and computational complexity, and no straightforward DL solution equally suits all types of data and applications. The code and HistoArtifacts dataset can be found online at Github and Zenodo, respectively.

CMN: a co-designed neural architecture search for efficient computing-in-memory-based mixture-of-experts

Artificial intelligence (AI) has experienced substantial advancements recently, notably with the advent of large-scale language models (LLMs) employing mixture-of-experts (MoE) techniques, exhibiting human-like cognitive skills. As a promising hardware solution for edge MoE implementations, the computing-in-memory (CIM) architecture collocates memory and computing within a single device, significantly reducing the data movement and the associated energy consumption. However, due to diverse edge application scenarios and constraints, determining the optimal network structures for MoE, such as the expert’s location, quantity, and dimension on CIM systems remains elusive. To this end, we introduce a software-hardware co-designed neural architecture search (NAS) framework, CIM-based MoE NAS (CMN), focusing on identifying a high-performing MoE structure under specific hardware constraints. The results of the NYUD-v2 dataset segmentation on the RRAM (SRAM) CIM system reveal that CMN can discover optimized MoE configurations under energy, latency, and performance constraints, achieving 29.67× (43.10×) energy savings, 175.44×(109.89×) speedup, and 12.24× smaller model size compared to the baseline MoE-enabled Visual Transformer, respectively. This co-design opens up an avenue toward high-performance MoE deployments in edge CIM systems.

CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval

The first-stage retrieval aims to retrieve a subset of candidate documents from a huge collection both effectively and efficiently. Since various matching patterns can exist between queries and relevant documents, previous work tries to combine multiple retrieval models to find as many relevant results as possible. The constructed ensembles, whether learned independently or jointly, do not care which component model is more suitable to an instance during training. Thus, they cannot fully exploit the capabilities of different types of retrieval models in identifying diverse relevance patterns. Motivated by this observation, in this paper, we propose a Mixture-of-Experts (MoE) model consisting of representative matching experts and a novel competitive learning mechanism to let the experts develop and enhance their expertise during training. Specifically, our MoE model shares the bottom layers to learn common semantic representations and uses differently structured upper layers to represent various types of retrieval experts. Our competitive learning mechanism has two stages: (1) a standardized learning stage to train the experts equally to develop their capabilities to conduct relevance matching; (2) a specialized learning stage where the experts compete with each other on every training instance and get rewards and updates according to their performance to enhance their expertise on certain types of samples. Experimental results on retrieval benchmark datasets show that our method significantly outperforms the state-of-the-art baselines in the in-domain and out-of-domain settings.

MicroBERT: Distilling MoE-Based Knowledge from BERT into a Lighter Model

Natural language-processing tasks have been improved greatly by large language models (LLMs). However, numerous parameters make their execution computationally expensive and difficult on resource-constrained devices. For this problem, as well as maintaining accuracy, some techniques such as distillation and quantization have been proposed. Unfortunately, current methods fail to integrate model pruning with downstream tasks and overlook sentence-level semantic modeling, resulting in reduced efficiency of distillation. To alleviate these limitations, we propose a novel distilled lightweight model for BERT named MicroBERT. This method can transfer the knowledge contained in the “teacher” BERT model to a “student” BERT model. The sentence-level feature alignment loss (FAL) distillation mechanism, guided by Mixture-of-Experts (MoE), captures comprehensive contextual semantic knowledge from the “teacher” model to enhance the “student” model’s performance while reducing its parameters. To make the outputs of “teacher” and “student” models comparable, we introduce the idea of a generative adversarial network (GAN) to train a discriminator. Our experimental results based on four datasets show that all steps of our distillation mechanism are effective, and the MicroBERT (101.14%) model outperforms TinyBERT (99%) by 2.24% in terms of average distillation reductions in various tasks on the GLUE dataset.

Addressing environmental stochasticity in reconfigurable intelligent surface aided unmanned aerial vehicle networks: Multi-task deep reinforcement learning based optimization for physical layer security

Deep Reinforcement Learning (DRL) is a powerful tool for optimizing communications in reconfigurable intelligent surface (RIS)-assisted millimeter-wave unmanned aerial vehicle (UAV) systems, particularly for ensuring physical layer security by dynamically maximizing signal strength for legitimate users while minimizing leakage towards eavesdroppers. However, existing approaches often overfit the DRL agent to a non-stochastic environment, where the trajectories of users and eavesdroppers are fixed. This results in a single-task DRL agent that cannot generalize beyond the training environment/task. In response, this study introduces a novel multi-task DRL framework to address this limitation. Firstly, recognizing that not all tasks share the same solution, we proposed a reward-driven clustering method to group similar tasks based on their reward scores. This enables the training of a shared multi-task DRL agent for each cluster. Each cluster can be treated as an expertise, and the cluster-specific multi-task DRL agents are considered the 'experts'. Meanwhile, we leverage federated learning (FL) to speed up the training of the shared multi-task DRL agent. By combining these two techniques, our reward-driven clustered FL autonomously groups similar tasks together rapidly. Simulation results demonstrate a reduction in the number of agents from 50 to 10 specialized 'experts', while achieving a 10X speedup compared to conventional single-task DRL training. Furthermore, the study presents a mixture of experts (MoE) model that unifies these 'experts' into a single model. The MoE is trained to select the most suitable 'expert' when encountering new tasks. Our MoE exhibits robustness by effectively generalizing to 50 novel tasks without fine-tuning.

Efficiently approaching vertical federated learning by combining data reduction and conditional computation techniques

In this paper, a framework based on a sparse Mixture of Experts (MoE) architecture is proposed for the federated learning and application of a distributed classification model in domains (like cybersecurity and healthcare) where different parties of the federation store different subsets of features for a number of data instances. The framework is designed to limit the risk of information leakage and computation/communication costs in both model training (through data sampling) and application (leveraging the conditional-computation abilities of sparse MoEs). Experiments on real data have shown the proposed approach to ensure a better balance between efficiency and model accuracy, compared to other VFL-based solutions. Notably, in a real-life cybersecurity case study focused on malware classification (the KronoDroid dataset), the proposed method surpasses competitors even though it utilizes only 50% and 75% of the training set, which is fully utilized by the other approaches in the competition. This method achieves reductions in the rate of false positives by 16.9% and 18.2%, respectively, and also delivers satisfactory results on the other evaluation metrics. These results showcase our framework’s potential to significantly enhance cybersecurity threat detection and prevention in a collaborative yet secure manner.

Efficient Inference Offloading for Mixture-of-Experts Large Language Models in Internet of Medical Things

Despite recent significant advancements in large language models (LLMs) for medical services, the deployment difficulties of LLMs in e-healthcare hinder complex medical applications in the Internet of Medical Things (IoMT). People are increasingly concerned about e-healthcare risks and privacy protection. Existing LLMs face difficulties in providing accurate medical questions and answers (Q&As) and meeting the deployment resource demands in the IoMT. To address these challenges, we propose MedMixtral 8x7B, a new medical LLM based on the mixture-of-experts (MoE) architecture with an offloading strategy, enabling deployment on the IoMT, improving the privacy protection for users. Additionally, we find that the significant factors affecting latency include the method of device interconnection, the location of offloading servers, and the speed of the disk.

Solving Table-based fact verification with mixture of logical reasoning

Table-based fact verification (TFV) is a binary classification task that requires understanding and reasoning about both table and text. This task poses many challenges, such as table parsing, text comprehension, and numerical reasoning. However, existing methods tend to depend solely on pre-trained models for tables, treating all types of reasoning equally and disregarding the importance of identifying logic types in inference process. In this regard, we propose MoETFV, an efficient and explanatory approach to solving TFV, which is based on a Mixture-of-Experts (MoE) framework. This approach can detect the underlying logic types of statements and leverage multiple independent experts to emulate diverse logical reasoning. It consists of one shared expert for general semantic understanding and several specific experts with distinct responsibilities for different logical inferences. Moreover, the practical applications of the MoE method in TFV are thoroughly investigated. This model doesn’t necessitate any table pre-trained models, and aligns closely with human cognitive processes in addressing such issues. Experimental results demonstrate the innovation and feasibility of the proposed approach.

EMPT: a sparsity Transformer for EEG-based motor imagery recognition.

Transformer network is widely emphasized and studied relying on its excellent performance. The self-attention mechanism finds a good solution for feature coding among multiple channels of electroencephalography (EEG) signals. However, using the self-attention mechanism to construct models on EEG data suffers from the problem of the large amount of data required and the complexity of the algorithm. We propose a Transformer neural network combined with the addition of Mixture of Experts (MoE) layer and ProbSparse Self-attention mechanism for decoding the time-frequency-spatial domain features from motor imagery (MI) EEG of spinal cord injury patients. The model is named as EEG MoE-Prob-Transformer (EMPT). The common spatial pattern and the modified s-transform method are employed for achieving the time-frequency-spatial features, which are used as feature embeddings to input the improved transformer neural network for feature reconstruction, and then rely on the expert model in the MoE layer for sparsity mapping, and finally output the results through the fully connected layer. EMPT achieves an accuracy of 95.24% on the MI EEG dataset for patients with spinal cord injury. EMPT has also achieved excellent results in comparative experiments with other state-of-the-art methods. The MoE layer and ProbSparse Self-attention inside the EMPT are subjected to visualisation experiments. The experiments prove that sparsity can be introduced to the Transformer neural network by introducing MoE and kullback-leibler divergence attention pooling mechanism, thereby enhancing its applicability on EEG datasets. A novel deep learning approach is presented for decoding EEG data based on MI.

Using Mixture of Experts to accelerate dataset distillation

Recently, large datasets have become increasingly necessary for most deep learning tasks, however, large datasets may bring some problems, such as disk storage and huge computational expense. Dataset distillation is an emerging field that aims to synthesize a small dataset from the original dataset, then a random model trained on the distillation dataset can achieve comparable performances to the same architecture model trained on the original dataset. Matching Training Trajectories (MTT) achieves a leading performance in this field, but it needs to pre-train 200 expert models before the formal distillation process, which is called buffer process. In this paper, we propose a new method to reduce the consumed time of buffer process. Concretely, we use Mixture of Experts (MoE) to train several expert models parallelly in buffer process. The experiments show our method can achieve a speedup of up to approximately 4∼8× in buffer process with getting comparable distillation performances.

Development and validation of a deep learning-based model to predict response and survival of T790M mutant non-small cell lung cancer patients in early clinical phase trials using electronic medical record and pharmacokinetic data.

Epidermal growth factor receptor (EGFR) T790M mutation is the standard predictive biomarker for third-generation epidermal growth factor receptor tyrosine kinase inhibitor (EGFR-TKI) treatment. While not all T790M-positive patients respond to third-generation EGFR-TKIs and have a good prognosis, it necessitates novel tools to supplement EGFR genotype detection for predicting efficacy and stratifying EGFR-mutant patients with various prognoses. Mixture-of-experts (MoE) is designed to disassemble a large model into many small models. Meanwhile, it is also a model ensembling method that can better capture multiple patterns of intrinsic subgroups of enrolled patients. Therefore, the combination of MoE and Cox algorithm has the potential to predict efficacy and stratify survival in non-small cell lung cancer (NSCLC) patients with EGFR mutations. We utilized the electronic medical record (EMR) and pharmacokinetic parameters of 326 T790M-mutated NSCLC patients, including 283 patients treated with Abivertinib in phase I (n=177, for training) and II (n=106, for validation) clinical trials and an additional validation cohort 2 comprising 43 patients treated with BPI-7711. Furthermore, 18 patients underwent whole-exome sequencing for biological interpretation of CoxMoE. We evaluated the predictive performance for therapeutic response using the area under the curve (AUC) and the Concordance index (C-index) for progression-free survival (PFS). CoxMoE exhibited AUCs of 0.73-0.83 for predicting efficacy defined by best overall response (BoR) and achieved C-index values of 0.64-0.65 for PFS prediction in training and validating cohorts. The PFS of 198 patients with a low risk [median, 6.0 (range, 1.0-23.3) months in the abivertinib treated cohort; median 16.5 (range, 1.4-27.4) months in BPI-7711 treated cohort] of being non-responder increased by 43% [hazard ratio (HR), 0.56; 95% confidence interval (CI), 0.40-0.78; P=0.0013] and 50% (HR, 0; 95% CI, 0-0; P=0.01) compared to those at high-risk [median, 4.2 (range, 1.0-35) months in the abivertinib treated cohort; median, 11.0 (range, 1.4-25.1) months in BPI-7711 treated cohort]. Additionally, activated partial thromboplastin time (APTT), creatinine clearance (Ccr), monocyte, and steady-state plasma trough concentration utilited to construct model were found significantly associated with drug resistance and aggressive tumor pathways. A robust correlation was observed between APTT and Ccr with PFS (log-rank test; P<0.01) and treatment response (Wilcoxon test; P<0.05), respectively. CoxMoE offers a valuable approach for patient selection by forecasting therapeutic response and PFS utilizing laboratory tests and pharmacokinetic parameters in the setting of early-phase clinical trials. Simultaneously, CoxMoE could predict the efficacy of third-generation EGFR-TKI non-invasively for T790M-positive NSCLC patients, thereby complementing existing EGFR genotype detection.

A novel deep Siamese framework for burned area mapping Leveraging mixture of experts

Due to the complexity of the areas and the diversity of the objects, traditional Burned Area Mapping (BAM) methods cannot provide promising results. Moreover, these methods focus on additional processing to improve their results, which is time-consuming and complex. Therefore, an advanced framework is needed to achieve accurate results in burned area mapping. In this context, this study proposes a novel Siamese-based mixture of expert networks for burned area mapping, which takes advantage of a mixture of experts (MoE) and position and channel attention mechanisms for deep feature generation. The proposed framework has two deep feature extractor channels to account for the bi-temporal multispectral pre- and post-fire input datasets. To evaluate the performance of the SMoE model, three multispectral Sentinel-2 were used in different countries. The results were compared with other advanced machine and deep learning models, including Light Gradient Boosting Machine (LGBM), Extreme Gradient Boosting (XGBoost), 2D-Siamse Convolutional Neural Network (2D-SCNN), and 3D-Siamse Convolutional Neural Network (3D-SCNN). The result of the burned mapping shows that the proposed model has a high effectiveness compared to other models, as it provides an average accuracy of more than 98% and 0.91 b y overall accuracy (OA) and kappa coefficient (KC) indices, respectively.

MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

The application of mixture-of-experts (MoE) is gaining popularity due to its ability to improve model's performance. In an MoE structure, the gate layer plays a significant role in distinguishing and routing input features to different experts. This enables each expert to specialize in processing their corresponding sub-tasks. However, the gate's routing mechanism also gives rise to "narrow vision": the individual MoE's expert fails to use more samples in learning the allocated subtask, which in turn limits the MoE to further improve its generalization ability. To effectively address this, we propose a method called Mixture-of-Distilled-Expert (MoDE), which applies moderate mutual distillation among experts to enable each expert to pick up more features learned by other experts and gain more accurate perceptions on their allocated sub-tasks. We conduct plenty experiments including tabular, NLP and CV datasets, which shows MoDE's effectiveness, universality and robustness. Furthermore, we develop a parallel study through innovatively constructing "expert probing", to experimentally prove why MoDE works: moderate distilling knowledge from other experts can improve each individual expert's test performances on their assigned tasks, leading to MoE's overall performance improvement.

Efficient Deweahter Mixture-of-Experts with Uncertainty-Aware Feature-Wise Linear Modulation

The Mixture-of-Experts (MoE) approach has demonstrated outstanding scalability in multi-task learning including low-level upstream tasks such as concurrent removal of multiple adverse weather effects. However, the conventional MoE architecture with parallel Feed Forward Network (FFN) experts leads to significant parameter and computational overheads that hinder its efficient deployment. In addition, the naive MoE linear router is suboptimal in assigning task-specific features to multiple experts which limits its further scalability. In this work, we propose an efficient MoE architecture with weight sharing across the experts. Inspired by the idea of linear feature modulation (FM), our architecture implicitly instantiates multiple experts via learnable activation modulations on a single shared expert block. The proposed Feature Modulated Expert (FME) serves as a building block for the novel Mixture-of-Feature-Modulation-Experts (MoFME) architecture, which can scale up the number of experts with low overhead. We further propose an Uncertainty-aware Router (UaR) to assign task-specific features to different FM modules with well-calibrated weights. This enables MoFME to effectively learn diverse expert functions for multiple tasks. The conducted experiments on the multi-deweather task show that our MoFME outperforms the state-of-the-art in the image restoration quality by 0.1-0.2 dB while saving more than 74% of parameters and 20% inference time over the conventional MoE counterpart. Experiments on the downstream segmentation and classification tasks further demonstrate the generalizability of MoFME to real open-world applications.

HDformer: A Higher-Dimensional Transformer for Detecting Diabetes Utilizing Long-Range Vascular Signals

Diabetes mellitus is a global concern, and early detection can prevent serious complications. 50% of those with diabetes live undiagnosed, disproportionately afflicting low-income groups. Non-invasive methods have emerged for timely detection; however, their limited accuracy constrains clinical usage. In this research, we present a novel Higher Dimensional Transformer (HDformer), the first Transformer-based architecture which utilizes long-range photoplethysmography (PPG) to detect diabetes. The long-range PPG maximizes signal contextual information when compared to the less-than 30 second signals commonly used in existing research. To increase the computational efficiency of HDformer’s long-range processing, a new attention module, Time Square Attention (TSA), is invented to achieve linear computational complexity with respect to the token volume while retaining the local/global dependencies. TSA converts the 1D inputs into 2D representations, grouping the adjacent points into a single 2D token. It then generates dynamic patches and feeds them into a gated mixture-of-experts (MoE) network, optimizing the learning on different attention areas. HDformer achieves state-of-the-art results (sensitivity 98.4, accuracy 97.3, specificity 92.8, AUC 0.929) on the standard MIMIC-III dataset, surpassing existing research. Furthermore, we develop an end-to-end solution where a low-cost wearable is prototyped to connect with the HDformer in the Cloud via a mobile app. This scalable, convenient, and affordable approach provides instantaneous detection and continuous monitoring for individuals. It aids doctors in easily screening for diabetes and safeguards underprivileged communities. The enhanced versatility of HDformer allows for efficient processing and learning of long-range signals in general one-dimensional time-series sequences, particularly for all biomedical waveforms.