Foundation Model Research Articles

MODNO: Multi-Operator learning with Distributed Neural Operators

The study of operator learning involves the utilization of neural networks to approximate operators. Traditionally, the focus has been on single-operator learning (SOL). However, recent advances have rapidly expanded this to include the approximation of multiple operators using foundation models equipped with millions or billions of trainable parameters, leading to the research of multi-operator learning (MOL). In this paper, we present a novel distributed training approach aimed at enabling a single neural operator with significantly fewer parameters to effectively tackle multi-operator learning challenges, all without incurring additional average costs. Our method is applicable to various neural operators, such as the Deep Operator Neural Networks (DON). The core idea is to independently learn the output basis functions for each operator using its dedicated data, while simultaneously centralizing the learning of the input function encoding shared by all operators using the entire dataset. Through a systematic study of five numerical examples, we compare the accuracy and cost of training a single neural operator for each operator independently versus training a MOL model using our proposed method. Our results demonstrate enhanced efficiency and satisfactory accuracy. Moreover, our approach illustrates that some operators with limited data can be more effectively constructed with the aid of data from analogous operators through MOL learning. This highlights another MOL’s potential to bolster operator learning.

XAI-Based Clinical Decision Support Systems: A Systematic Review

With increasing electronic medical data and the development of artificial intelligence, clinical decision support systems (CDSSs) assist clinicians in diagnosis and prescription. Traditional knowledge-based CDSSs follow an accumulated medical knowledgebase and a predefined rule system, which clarifies the decision-making process; however, maintenance cost issues exist in the medical data quality control and standardization processes. Non-knowledge-based CDSSs utilize vast amounts of data and algorithms to effectively make decisions; however, the deep learning black-box problem causes unreliable results. EXplainable Artificial Intelligence (XAI)-based CDSSs provide valid rationales and explainable results. These systems ensure trustworthiness and transparency by showing the recommendation and prediction result process using explainable techniques. However, existing systems have limitations, such as the scope of data utilization and the lack of explanatory power of AI models. This study proposes a new XAI-based CDSS framework to address these issues; introduces resources, datasets, and models that can be utilized; and provides a foundation model to support decision-making in various disease domains. Finally, we propose future directions for CDSS technology and highlight societal issues that need to be addressed to emphasize the potential of CDSSs in the future.

Efficient greenhouse segmentation with visual foundation models: achieving more with fewer samples

Introduction: The Vision Transformer (ViT) model, which leverages self-supervised learning, has shown exceptional performance in natural image segmentation, suggesting its extensive potential in visual tasks. However, its effectiveness diminishes in remote sensing due to the varying perspectives of remote sensing images and unique optical properties of features like the translucency of greenhouses. Additionally, the high cost of training Visual Foundation Models (VFMs) from scratch for specific scenes limits their deployment.Methods: This study investigates the feasibility of rapidly deploying VFMs on new tasks by using embedding vectors generated by VFMs as prior knowledge to enhance traditional segmentation models’ performance. We implemented this approach to improve the accuracy and robustness of segmentation with the same number of trainable parameters. Comparative experiments were conducted to evaluate the efficiency and effectiveness of this method, especially in the context of greenhouse detection and management.Results: Our findings indicate that the use of embedding vectors facilitates rapid convergence and significantly boosts segmentation accuracy and robustness. Notably, our method achieves or exceeds the performance of traditional segmentation models using only about 40% of the annotated samples. This reduction in the reliance on manual annotation has significant implications for remote sensing applications.Discussion: The application of VFMs in remote sensing tasks, particularly for greenhouse detection and management, demonstrated enhanced segmentation accuracy and reduced dependence on annotated samples. This method adapts more swiftly to different lighting conditions, enabling more precise monitoring of agricultural resources. Our study underscores the potential of VFMs in remote sensing tasks and opens new avenues for the expansive application of these models in diverse downstream tasks.

Facial recognition models for identifying genetic syndromes associated with pulmonary stenosis in children.

Williams-Beuren syndrome, Noonan syndrome, and Alagille syndrome are common types of genetic syndromes (GSs) characterized by distinct facial features, pulmonary stenosis, and delayed growth. In clinical practice, differentiating these three GSs remains a challenge. Facial gestalts serve as a diagnostic tool for recognizing Williams-Beuren syndrome, Noonan syndrome, and Alagille syndrome. Pretrained foundation models (PFMs) can be considered the foundation for small-scale tasks. By pretraining with a foundation model, we propose facial recognition models for identifying these syndromes. A total of 3297 (n = 1666) facial photos were obtained from children diagnosed with Williams-Beuren syndrome (n = 174), Noonan syndrome (n = 235), and Alagille syndrome (n = 51), and from children without GSs (n = 1206). The photos were randomly divided into five subsets, with each syndrome and non-GS equally and randomly distributed in each subset. The proportion of the training set and the test set was 4:1. The ResNet-100 architecture was employed as the backbone model. By pretraining with a foundation model, we constructed two face recognition models: one utilizing the ArcFace loss function, and the other employing the CosFace loss function. Additionally, we developed two models using the same architecture and loss function but without pretraining. The accuracy, precision, recall, and F1 score of each model were evaluated. Finally, we compared the performance of the facial recognition models to that of five pediatricians. Among the four models, ResNet-100 with a PFM and CosFace loss function achieved the best accuracy (84.8%). Of the same loss function, the performance of the PFMs significantly improved (from 78.5% to 84.5% for the ArcFace loss function, and from 79.8% to 84.8% for the CosFace loss function). With and without the PFM, the performance of the CosFace loss function models was similar to that of the ArcFace loss function models (79.8% vs 78.5% without PFM; 84.8% vs 84.5% with PFM). Among the five pediatricians, the highest accuracy (0.700) was achieved by the senior-most pediatrician with genetics training. The accuracy and F1 scores of the pediatricians were generally lower than those of the models. A facial recognition-based model has the potential to improve the identification of three common GSs with pulmonary stenosis. PFMs might be valuable for building screening models for facial recognition. Key messages What is already known on this topic: Early identification of genetic syndromes (GSs) is crucial for the management and prognosis of children with pulmonary stenosis (PS). Facial phenotyping with convolutional neural networks (CNNs) often requires large-scale training data, limiting its usefulness for GSs. What this study adds: We successfully built multi-classification models based on face recognition using a CNN to accurately identify three common PS-associated GSs. ResNet-100 with a pretrained foundation model (PFM) and CosFace loss function achieved the best accuracy (84.8%). Pretrained with the foundation model, the performance of the models significantly improved, although the impact of the type of loss function appeared to be minimal. How this study might affect research, practice, or policy: A facial recognition-based model has the potential to improve the identification of GSs in children with PS. The PFM might be valuable for building identification models for facial detection.

Protein Set Transformer: A protein-based genome language model to power high diversity viromics.

Exponential increases in microbial and viral genomic data demand transformational advances in scalable, generalizable frameworks for their interpretation. Standard homology-based functional analyses are hindered by the rapid divergence of microbial and especially viral genomes and proteins that significantly decreases the volume of usable data. Here, we present Protein Set Transformer (PST), a protein-based genome language model that models genomes as sets of proteins without considering sparsely available functional labels. Trained on >100k viruses, PST outperformed other homology- and language model-based approaches for relating viral genomes based on shared protein content. Further, PST demonstrated protein structural and functional awareness by clustering capsid-fold-containing proteins with known capsid proteins and uniquely clustering late gene proteins within related viruses. Our data establish PST as a valuable method for diverse viral genomics, ecology, and evolutionary applications. We posit that the PST framework can be a foundation model for microbial genomics when trained on suitable data.

Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration

Visual Question Answering (VQA) has recently emerged as a potential research domain, captivating the interest of many in the field of artificial intelligence and computer vision. Despite the prevalence of approaches in English, there is a notable lack of systems specifically developed for certain languages, particularly Vietnamese. This study aims to bridge this gap by conducting comprehensive experiments on the Vietnamese Visual Question Answering (ViVQA) dataset, demonstrating the effectiveness of our proposed model. In response to community interest, we have developed a model that enhances image representation capabilities, thereby improving overall performance in the ViVQA system. Therefore, we propose AViVQA-TranConI (Advancing Vietnamese Visual Question Answering with Transformer and Convolutional Integration). AViVQA-TranConI integrates the Bootstrapping Language-Image Pre-training with frozen unimodal models (BLIP-2) and the convolutional neural network EfficientNet to extract and process both local and global features from images. This integration leverages the strengths of transformer-based architectures for capturing comprehensive contextual information and convolutional networks for detailed local features. By freezing the parameters of these pre-trained models, we significantly reduce the computational cost and training time, while maintaining high performance. This approach significantly improves image representation and enhances the performance of existing VQA systems. We then leverage a multi-modal fusion module based on a general-purpose multi-modal foundation model (BEiT-3) to fuse the information between visual and textual features. Our experimental findings demonstrate that AViVQA-TranConI surpasses competing baselines, achieving promising performance. This is particularly evident in its accuracy of 71.04% on the test set of the ViVQA dataset, marking a significant advancement in our research area. The code is available at https://github.com/nngocson2002/ViVQA.

Leaf only SAM: A segment anything pipeline for zero-shot automated leaf segmentation

Segment Anything Model (SAM) is a new “foundation model” that can be used as a zero-shot object segmentation method with the use of either guide prompts such as bounding boxes, polygons, or points. Alternatively, additional post processing steps can be used to identify objects of interest after segmenting everything in an image. Here a method is presented using segment anything together with a series of post processing steps to segment potato leaves, called Leaf Only SAM. The advantage of this proposed method is that it does not require any training data to produce its results so has many applications across the field of plant phenotyping where there is limited high quality annotated data available. The performance of Leaf Only SAM is compared to a Mask R-CNN model which has been fine-tuned on a small novel potato leaf dataset. On the evaluation dataset, Leaf Only SAM finds an average recall of 73.1 and an average precision of 73.9, compared to recall of 87.6 and precision of 84.4 for Mask R-CNN. Leaf Only SAM does not perform better than the fine-tuned Mask R-CNN model on the potato leaf dataset, but the SAM based model does not require any extra training or annotation. This shows there is potential to use SAM as a zero-shot classifier with the addition of post processing steps.

Integrated vision language and foundation model for automated estimation of building lowest floor elevation

Integrated vision language and foundation model for automated estimation of building lowest floor elevation

Visual Tuning

Fine-tuning visual models has been widely shown promising performance on many downstream visual tasks. With the surprising development of pre-trained visual foundation models, visual tuning jumped out of the standard modus operandi that fine-tunes the whole pre-trained model or just the fully connected layer. Instead, recent advances can achieve superior performance than full-tuning the whole pre-trained parameters by updating far fewer parameters, enabling edge devices and downstream applications to reuse the increasingly large foundation models deployed on the cloud. With the aim of helping researchers get the full picture and future directions of visual tuning, this survey characterizes a large and thoughtful selection of recent works, providing a systematic and comprehensive overview of existing work and models. Specifically, it provides a detailed background of visual tuning and categorizes recent visual tuning techniques into five groups: fine-tuning, prompt tuning, adapter tuning, parameter tuning, and remapping tuning. Meanwhile, it offers some exciting research directions for prospective pre-training and various interactions in visual tuning.

The Jaynes–Cummings model: 60 years and still counting

2023 marked the 60th anniversary of the Jaynes–Cummings model, a foundational model in quantum optics. Over the years, its importance has expanded beyond traditional light–matter interaction systems, such as cavity QED. This special issue presents a collection of articles that showcase the evolution of the model’s applications, blending traditional topics with contemporary developments.

Foundation models for bioinformatics

AbstractTransformer‐based foundation models such as ChatGPTs have revolutionized our daily life and affected many fields including bioinformatics. In this perspective, we first discuss about the direct application of textual foundation models on bioinformatics tasks, focusing on how to make the most out of canonical large language models and mitigate their inherent flaws. Meanwhile, we go through the transformer‐based, bioinformatics‐tailored foundation models for both sequence and non‐sequence data. In particular, we envision the further development directions as well as challenges for bioinformatics foundation models.

Extracting and Utilizing Multimodal Microscopy Datasets of Images and Text with Foundation Models

Extracting and Utilizing Multimodal Microscopy Datasets of Images and Text with Foundation Models

Research on the Reinforcement Characteristics of Thick Cushion Layer and Rigid Pile Composite Foundation

The rigid pile composite foundation method is the most commonly used method for strengthening weak soil foundations. In this method, piles usually need to pass through weak soil layers, and the pile end falls on a bearing layer with good bearing capacity. Under existing technical conditions, the thicker the weak soil layer, the longer the pile body, and the more difficult it is to ensure the construction quality of the pile. In response to this issue, some scholars have adopted the rigid pile composite foundation method with a thick cushion layer for reinforcement treatment. This article uses PLAXIS 3D (V20.04.00.790) software to establish a finite element model of rigid pile composite foundation with a thick cushion layer and simulate the process of foundation reinforcement. The influence of parameters such as thickness, compression modulus, and shear strength index of the cushion layer on foundation settlement and pile–soil stress distribution is studied, and the reasonable range of these parameters is analyzed under the condition of considering reinforcement effect. Through comparative analysis, it can be concluded that for deep and weak soil areas, the thickness of the cushion layer can range from 0.5 to 2.6. The thickness and compressive modulus of the cushion layer have a significant impact on the settlement of the foundation, the pile–soil stress ratio, and the stress of the pile body, while the shear strength index of the cushion layer has a relatively small impact on these parameters. Reasonably selecting the geometric and mechanical parameters of the cushion layer can effectively reduce stress concentration at the pile top and better play the role of the soil between piles.

Implementing the predictor-corrector approach to examine the thermo-solutal convection for buongiorno eyring-powell nanofluid model with squeezed microcantilivier surface

The Buongiorno nanofluid model serves as a foundational model for theoretical and experimental research in the field of nanofluids, and this model can be applied to various engineering problems, including heat exchangers, biomedical applications, solar collectors, nuclear reactors, and different cooling systems in the automotive and electronics industries. The use of nanofluids in the Eyring-Powell fluid over a squeezed sensing surfaces is a vital area of research for the design and improvement of microfluidic devices and sensors, which find widespread usage in industrial and medical applications. The main purpose of this work is to note the augmentation in thermal conductivity by using the Buongiorno nanofluid model along with the natural convective flow of non-Newtonian Eyring-Powell fluid that is moving on the squeezed sensory surface along with slip velocity. The viscosity of a fluid is assumed to be dependent on temperature. The Cattaneo-Christov heat and mass flux and chemical reaction are assumed to determine the combined effect of heat and mass transport. The governing model of equations is in the form of dimensional partial differential equations, and we have to convert these equations into ordinary differential equations; therefore, a set of similarity transformations is adopted for making the equations into dimensionless form. The numerical results were obtained by adopting the predictor and corrector multistep method, namely the ‘Adams-Bashforth’ technique, in Matlab software. The use of natural convective flow enhances the velocity region. The velocity slip parameter increases the velocity of the fluid, whereas the viscosity parameter drops the velocity profile. The thermophoresis and Brownian motion parameters are the sources of the increment in the temperature region. The thermal relaxation and solutal relaxation parameters are the sources of decline in the temperature region. The squeezing parameter is the source of reduction in the skin friction, whereas the result indicates that the fluid parameter, Grashof number, and viscosity coefficients increase the skin friction.

A foundation model for clinical-grade computational pathology and rare cancers detection.

The analysis of histopathology images with artificial intelligence aims to enable clinical decision support systems and precision medicine. The success of such applications depends on the ability to model the diverse patterns observed in pathology images. To this end, we present Virchow, the largest foundation model for computational pathology to date. In addition to the evaluation of biomarker prediction and cell identification, we demonstrate that a large foundation model enables pan-cancer detection, achieving 0.95 specimen-level area under the (receiver operating characteristic) curve across nine common and seven rare cancers. Furthermore, we show that with less training data, the pan-cancer detector built on Virchow can achieve similar performance to tissue-specific clinical-grade models in production and outperform them on some rare variants of cancer. Virchow's performance gains highlight the value of a foundation model and open possibilities for many high-impact applications with limited amounts of labeled training data.

Can language models handle recursively nested grammatical structures? A case study on comparing models and humans

Abstract How should we compare the capabilities of language models (LMs) and humans? In this paper, I draw inspiration from comparative psychology to highlight challenges in these comparisons. I focus on a case study: processing of recursively nested grammatical structures. Prior work suggests that LMs cannot process these structures as reliably as humans can. However, the humans were provided with instructions and substantial training, while the LMs were evaluated zero-shot. I therefore match the evaluation more closely. Providing large LMs with a simple prompt—with substantially less content than the human training—allows the LMs to consistently outperform the human results, even in more deeply nested conditions than were tested with humans. Furthermore, the effects of prompting are robust to the particular structures and vocabulary used in the prompt. Finally, reanalyzing the existing human data suggests that the humans may not perform above chance at the difficult structures initially. Thus, large LMs may indeed process recursively nested grammatical structures as reliably as humans, when evaluated comparably. This case study highlights how discrepancies in the evaluation methods can confound comparisons of language models and humans. I conclude by reflecting on the broader challenge of comparing human and model capabilities, and highlight an important difference between evaluating cognitive models and foundation models.

Two-Stage Analysis Method for the Mechanical Response of Adjacent Existing Tunnels Caused by Foundation Pit Excavation

With the advancement of urban underground space networks, there has been a rise in foundation pit projects near existing tunnels. The construction of these foundation pits adjacent to existing tunnels can result in soil disturbance and stress redistribution, leading to additional deformation and internal force within the tunnels. This paper delves into the two-stage analysis method, outlining the calculation of additional stress in the initial stage considering various engineering factors and the methods for determining tunnel displacement and internal force in the subsequent stage. Through an engineering example and numerical simulations, the theoretical calculations were validated. The maximum displacement generated by the tunnel is −4.85 mm and −5.10 mm, respectively. The maximum error is only 5.9%, which confirms the validity of the theoretical approach. The analysis demonstrates that incorporating the unloading model of the bottom and surrounding side walls of the foundation pit is essential when calculating additional stress in the first stage. Moreover, the presence of engineering dewatering and double-hole tunnels can counterbalance the additional stress, with deviations of only 4.4% and 2.5%, respectively. In the second stage, factoring in the shear action and lateral soil action in the foundation and tunnel model enhances the accuracy of stress mode representation (accuracy increased by 18.8% and 29.3%, respectively). Additionally, accounting for the buried depth effect of the tunnel, soil non-uniformity, and foundation nonlinearity helps prevent excessive foundation reactions.

Frequency analysis of open/closed polygonal channel aerostructures made of hybrid nanocomposite materials and supported by piecewise elastic foundation using GDQE technique

This research delves into the theoretical exploration of the generalized differential quadrature element (GDQE) method for analyzing the dynamic response and natural frequency of aerospace structures, specifically polygonal channels found in aircraft, satellites, and space vehicle components. These structures are critical in aerospace engineering, serving in various capacities such as airframe components and fluid conveyance systems. Emphasizing the practical significance of these structures, the study incorporates the influence of ground or support conditions, employing a two-parameter piecewise elastic foundation model. The investigated open or closed channels are characterized as laminated functionally graded hybrid nanocomposites, specifically denoted as FG-HNC, reinforced with Carbon nanotubes (CNTs) and Clay nanoparticles (CNP). Each composite layer is reinforced with randomly oriented and uniformly distributed tubes and particles. The homogenization of the hybrid composite domain is performed using the hierarchy Halpin-Tsai micromechanical rule. Utilizing the GDQE technique, the channels are discretized into plate elements, and motion equations for each plate element are calculated based on first-order shear deformation theory and Hamilton's principle. The resulting equations are then solved using the standard GDQ method, with subsequent application of appropriate continuity conditions at shared element boundaries. In this comprehensive study, the research scrutinizes the effects of structure dimensions, channel sides crank angles, composite characteristics, and boundary conditions on the free vibration behavior of the polygonal channels. By offering detailed insights into the vibrational characteristics of FG-HNC structures, this study contributes to the advancement of aerospace engineering design and analysis.

Overcoming data scarcity in biomedical imaging with a foundational multi-task model.

Foundational models, pretrained on a large scale, have demonstrated substantial success across non-medical domains. However, training these models typically requires large, comprehensive datasets, which contrasts with the smaller and more specialized datasets common in biomedical imaging. Here we propose a multi-task learning strategy that decouples the number of training tasks from memory requirements. We trained a universal biomedical pretrained model (UMedPT) on a multi-task database including tomographic, microscopic and X-ray images, with various labeling strategies such as classification, segmentation and object detection. The UMedPT foundational model outperformed ImageNet pretraining and previous state-of-the-art models. For classification tasks related to the pretraining database, it maintained its performance with only 1% of the original training data and without fine-tuning. For out-of-domain tasks it required only 50% of the original training data. In an external independent validation, imaging features extracted using UMedPT proved to set a new standard for cross-center transferability.

Multi-task learning for medical foundation models.

Multi-task learning for medical foundation models.