Foundation Model Research Articles

Few-shot adaptation of multi-modal foundation models: a survey

Multi-modal (vision-language) models, such as CLIP, are replacing traditional supervised pre-training models (e.g., ImageNet-based pre-training) as the new generation of visual foundation models. These models with robust and aligned semantic representations learned from billions of internet image-text pairs and can be applied to various downstream tasks in a zero-shot manner. However, in some fine-grained domains like medical imaging and remote sensing, the performance of multi-modal foundation models often leaves much to be desired. Consequently, many researchers have begun to explore few-shot adaptation methods for these models, gradually deriving three main technical approaches: (1) prompt-based methods, (2) adapter-based methods, and (3) external knowledge-based methods. Nevertheless, this rapidly developing field has produced numerous results without a comprehensive survey to systematically organize the research progress. Therefore, in this survey, we introduce and analyze the research advancements in few-shot adaptation methods for multi-modal models, summarizing commonly used datasets and experimental setups, and comparing the results of different methods. In addition, due to the lack of reliable theoretical support for existing methods, we derive the few-shot adaptation generalization error bound for multi-modal models. The theorem reveals that the generalization error of multi-modal foundation models is constrained by three factors: domain gap, model capacity, and sample size. Based on this, we propose three possible solutions from the following aspects: (1) adaptive domain generalization, (2) adaptive model selection, and (3) adaptive knowledge utilization.Kindly check and confirm the edit made in the title.The title is correct.

Foundation model for efficient biological discovery in single-molecule data.

Modern data-intensive techniques offer ever deeper insights into biology, but render the process of discovery increasingly complex. For example, exploiting the unique ability of single-molecule fluorescence microscopy (SMFM)1-5. to uncover rare but critical intermediates often demands manual inspection of time traces and iterative ad hoc approaches that are difficult to systematize. To facilitate systematic and efficient discovery from SMFM data, we introduce META-SiM, a transformer-based foundation model pre-trained on diverse SMFM analysis tasks. META-SiM achieves high performance-rivaling best-in-class algorithms-on a broad range of analysis tasks including trace selection, classification, segmentation, idealization, and stepwise photobleaching analysis. Additionally, the model produces high-dimensional embedding vectors that encapsulate detailed information about each trace, which the web-based META-SiM Projector (https://www.simol-projector.org) casts into lower-dimensional space for efficient whole-dataset visualization, labeling, comparison, and sharing. Combining this Projector with the objective metric of Local Shannon Entropy enables rapid identification of condition-specific behaviors, even if rare or subtle. As a result, by applying META-SiM to an existing single-molecule Förster resonance energy transfer (smFRET) dataset6, we discover a previously unobserved intermediate state in pre-mRNA splicing. META-SiM thus removes bottlenecks, improves objectivity, and both systematizes and accelerates biological discovery in complex single-molecule data.

Automated machine learning with interpretation: A systematic review of methodologies and applications in healthcare

AbstractMachine learning (ML) has achieved substantial success in performing healthcare tasks in which the configuration of every part of the ML pipeline relies heavily on technical knowledge. To help professionals with borderline expertise to better use ML techniques, Automated ML (AutoML) has emerged as a prospective solution. However, most models generated by AutoML are black boxes that are challenging to comprehend and deploy in healthcare settings. We conducted a systematic review to examine AutoML with interpretation systems for healthcare. We searched four databases (MEDLINE, EMBASE, Web of Science, and Scopus) complemented with seven prestigious ML conferences (AAAI, ACL, ICLR, ICML, IJCAI, KDD, and NeurIPS) that reported AutoML with interpretation for healthcare before September 1, 2023. We included 118 articles related to AutoML with interpretation in healthcare. First, we illustrated AutoML techniques used in the included publications, including automated data preparation, automated feature engineering, and automated model development, accompanied by a real‐world case study to demonstrate the advantages of AutoML over classic ML. Then, we summarized interpretation methods: feature interaction and importance, data dimensionality reduction, intrinsically interpretable models, and knowledge distillation and rule extraction. Finally, we detailed how AutoML with interpretation has been used for six major data types: image, free text, tabular data, signal, genomic sequences, and multi‐modality. To some extent, AutoML with interpretation provides effortless development and improves users' trust in ML in healthcare settings. In future studies, researchers should explore automated data preparation, seamless integration of automation and interpretation, compatibility with multi‐modality, and utilization of foundation models.

Evaluating the Utilities of Foundation Models in Single-cell Data Analysis.

Foundation Models (FMs) have made significant strides in both industrial and scientific domains. In this paper, we evaluate the performance of FMs for single-cell sequencing data analysis through comprehensive experiments across eight downstream tasks pertinent to single-cell data. Overall, the top FMs include scGPT, Geneformer, and CellPLM by considering model performances and user accessibility among ten single-cell FMs. However, by comparing these FMs with task-specific methods, we found that single-cell FMs may not consistently excel than task-specific methods in all tasks, which challenges the necessity of developing foundation models for single-cell analysis. In addition, we evaluated the effects of hyper-parameters, initial settings, and stability for training single-cell FMs based on a proposed scEval framework, and provide guidelines for pre-training and fine-tuning, to enhance the performances of single-cell FMs. Our work summarizes the current state of single-cell FMs, points to their constraints and avenues for future development, and offers a freely available evaluation pipeline to benchmark new models and improve method development.

A vision-language foundation model for the generation of realistic chest X-ray images.

The paucity of high-quality medical imaging datasets could be mitigated by machine learning models that generate compositionally diverse images that faithfully represent medical concepts and pathologies. However, large vision-language models are trained on natural images, and the diversity distribution of the generated images substantially differs from that of medical images. Moreover, medical language involves specific and semantically rich vocabulary. Here we describe a domain-adaptation strategy for large vision-language models that overcomes distributional shifts. Specifically, by leveraging publicly available datasets of chest X-ray images and the corresponding radiology reports, we adapted a latent diffusion model pre-trained on pairs of natural images and text descriptors to generate diverse and visually plausible synthetic chest X-ray images (as confirmed by board-certified radiologists) whose appearance can be controlled with free-form medical text prompts. The domain-adaptation strategy for the text-conditioned synthesis of medical images can be used to augment training datasets and is a viable alternative to the sharing of real medical images for model training and fine-tuning.

Comparative study on dynamic responses of ballasted and ballastless tracks at critical velocity

High-speed railways are becoming ubiquitous worldwide, demanding increased operational speeds. Yet, as trains near critical speeds, dynamic stress and embankment displacement escalate, jeopardizing safety. While significant strides have been made in studying these critical speeds, capturing the dynamic characteristics of all track responses remains elusive. This study presents a coupled train-track-ground model employing a 2.5D finite element approach integrated with a nonlinear soil model to investigate the influence of embankment and foundation properties on the critical speeds for both ballasted and ballastless tracks. The research resulted in the development of unified Dynamic Amplification Factor curves that consistently represent the dynamic behavior of various tracks as observed in multiple studies. Additionally, the applicability of simplified theoretical models for calculating critical speeds, such as the elastic half-space foundation and track-foundation models, was assessed. The findings suggest that the simplified models are suitable only under specific conditions. This research provides valuable perspectives on optimizing train speeds and ensuring the safety of diverse track types.

Comparative study of the foundation model of a 220 kV transmission line tower with different footing steps - Finite element analysis

Comparative study of the foundation model of a 220 kV transmission line tower with different footing steps - Finite element analysis

Dynamic Analysis of Force Redistribution in Small Pile Groups Foundation Subjected to Accidental Lateral Loads

Abstract Designing infrastructure always involves considering potential loads. Beyond anticipated loads, accidental ones can occur, as with the Pedamaran II bridge in Riau, where a ship collision damaged its foundation, breaking a pile and cracking the pile cap. However, this did not lead to structural failure, even though over 30% of the piles in the group were damaged. Initial hypotheses suggest that the use of bracing in the existing bridge might have helped redistribute forces. A numerical model was created using Opensees to simulate lateral loading on the shaft of an elevated pile cap, comparing the effects of bracing and no bracing using elastic elements in a simple 2x2 foundation model. The aim was to understand the correct modeling approach for bracing behavior and its impact on load response. Pushover analysis was performed on one of the shafts, with eccentric loading generating forces in both the x and y axes. Results showed that modeling bracing with elastic elements helped evenly distribute forces across rows of piles, with the greatest distribution occurring in the loaded row. The maximum moment occurred at the lowest bracing connection, highlighting the crucial role of bracing in load distribution, and significantly increasing lateral capacity by about 60%.

ScDiffusion: conditional generation of high-quality single-cell data using diffusion model.

Single-cell RNA sequencing (scRNA-seq) data are important for studying the laws of life at single-cell level. However, it is still challenging to obtain enough high-quality scRNA-seq data. To mitigate the limited availability of data, generative models have been proposed to computationally generate synthetic scRNA-seq data. Nevertheless, the data generated with current models are not very realistic yet, especially when we need to generate data with controlled conditions. In the meantime, diffusion models have shown their power in generating data with high fidelity, providing a new opportunity for scRNA-seq generation. In this study, we developed scDiffusion, a generative model combining the diffusion model and foundation model to generate high-quality scRNA-seq data with controlled conditions. We designed multiple classifiers to guide the diffusion process simultaneously, enabling scDiffusion to generate data under multiple condition combinations. We also proposed a new control strategy called Gradient Interpolation. This strategy allows the model to generate continuous trajectories of cell development from a given cell state. Experiments showed that scDiffusion could generate single-cell gene expression data closely resembling real scRNA-seq data. Also, scDiffusion can conditionally produce data on specific cell types including rare cell types. Furthermore, we could use the multiple-condition generation of scDiffusion to generate cell type that was out of the training data. Leveraging the Gradient Interpolation strategy, we generated a continuous developmental trajectory of mouse embryonic cells. These experiments demonstrate that scDiffusion is a powerful tool for augmenting the real scRNA-seq data and can provide insights into cell fate research. scDiffusion is openly available at the GitHub repository https://github.com/EperLuo/scDiffusion or Zenodo https://zenodo.org/doi/10.5281/zenodo.13268742.

MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation

The Segment Anything Model (SAM), a foundation model for general image segmentation, has demonstrated impressive zero-shot performance across numerous natural image segmentation tasks. However, SAM’s performance significantly declines when applied to medical images, primarily due to the substantial disparity between natural and medical image domains. To effectively adapt SAM to medical images, it is important to incorporate critical third-dimensional information, i.e., volumetric or temporal knowledge, during fine-tuning. Simultaneously, we aim to harness SAM’s pre-trained weights within its original 2D backbone to the fullest extent. In this paper, we introduce a modality-agnostic SAM adaptation framework, named as MA-SAM, that is applicable to various volumetric and video medical data. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments while preserving the majority of SAM’s pre-trained weights. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data. We comprehensively evaluate our method on five medical image segmentation tasks, by using 11 public datasets across CT, MRI, and surgical video data. Remarkably, without using any prompt, our method consistently outperforms various state-of-the-art 3D approaches, surpassing nnU-Net by 0.9%, 2.6%, and 9.9% in Dice for CT multi-organ segmentation, MRI prostate segmentation, and surgical scene segmentation respectively. Our model also demonstrates strong generalization, and excels in challenging tumor segmentation when prompts are used. Our code is available at: https://github.com/cchen-cc/MA-SAM.

Fine-Tuning SSL-Model to Enhance Detection of Cilioretinal Arteries on Colored Fundus Images.

Cilioretinal arteries are a common congenital anomaly of retinal blood supply. This paper presents a deep learning-based approach for the automated detection of a CRA from color fundus images. Leveraging the Vision Transformer architecture, a pre-trained model from RETFound was fine-tuned to transfer knowledge from a broader dataset to our specific task. An initial dataset of 85 was expanded to 170 images through data augmentation using self-supervised learning-driven techniques. To address the imbalance in the dataset and prevent overfitting, Focal Loss and Early Stopping were implemented. The model's performance was evaluated using a 70-30 split of the dataset for training and validation. The results showcase the potential of ophthalmic foundation models in enhancing detection of CRAs and reducing the effort required for labeling by retinal experts, as promising results could be achieved with only a small amount of training data through fine-tuning.

A Single Model for the Thermodynamics and Kinetics of Metal Exsolution from Perovskite Oxides.

Exsolution has emerged as an outstanding route for producing oxide-supported metal nanoparticles. For ABO3-perovskite oxides, various late transition-metal cations can be substituted into the lattice under oxidizing conditions and exsolved as metal nanoparticles after reduction. A consistent and comprehensive description of the point-defect thermodynamics and kinetics of this phenomenon is lacking, however. Herein, supported by hybrid density-functional-theory calculations, we propose a single model that explains diverse experimental observations, such as why substituent transition-metal cations (but not host cations) exsolve from perovskite oxides upon reduction; why different substituent transition-metal cations exsolve under different conditions; why the metal nanoparticles are embedded in the surface; why exsolution occurs surprisingly rapidly at relatively low temperatures; and why the reincorporation of exsolved species involves far longer times and much higher temperatures. Our model's foundation is that the substituent transition-metal cations are reduced to neutral species within the perovskite lattice as the Fermi level is shifted upward within the bandgap upon sample reduction. The calculations also indicate unconventional influences of oxygen vacancies and A-site vacancies. Our model thus provides a fundamental basis for improving existing, and creating new, exsolution-generated catalysts.

Carbon-14 labeled transformation of atrazine in soils: Comparison of superabsorbent hydrogel coating and technical material

Atrazine exhibits adverse effects on diverse organisms in both terrestrial and aquatic environments, even though it effectively targets specific organisms. This study employed superabsorbent hydrogels to coat 14C-atrazine coupled with a four-compartment model to determine the fate of this herbicide in three oxic soils over a 100-day incubation period. Mineralization of atrazine was limited in all soils, with rates remaining below 3.5 %. The encapsulation treatment reduced mineralization of atrazine in soil A and soil B. Bound residues ranged from 26.1 to 43.6 % at 100 d. The encapsulation treatment enhanced the degradation of atrazine and reduced the content of deethylatrazine in soil A, but significantly increased the content of deisopropylatrazine in soil A and hydroxyatrazine in soil C. Using the obtained data, we also constructed a four-compartment model to clarify the relationships among the parent compound, degradation products, bound residues, and mineralization. This model accurately fits the fate of atrazine in the present work. Additionally, the correlation study suggested that both soil parameters and superabsorbent hydrogels played significant roles in influencing atrazine transformation. These findings serve as a reference for evaluating the environmental impact of superabsorbent hydrogels in atrazine pollution reduction and offer a foundational model approach for a comprehensive understanding of organic pollutants.

Profiling cell identity and tissue architecture with single-cell and spatial transcriptomics.

Single-cell transcriptomics has broadened our understanding of cellular diversity and gene expression dynamics in healthy and diseased tissues. Recently, spatial transcriptomics has emerged as a tool to contextualize single cells in multicellular neighbourhoods and to identify spatially recurrent phenotypes, or ecotypes. These technologies have generated vast datasets with targeted-transcriptome and whole-transcriptome profiles of hundreds to millions of cells. Such data have provided new insights into developmental hierarchies, cellular plasticity and diverse tissue microenvironments, and spurred a burst of innovation in computational methods for single-cell analysis. In this Review, we discuss recent advancements, ongoing challenges and prospects in identifying and characterizing cell states and multicellular neighbourhoods. We discuss recent progress in sample processing, data integration, identification of subtle cell states, trajectory modelling, deconvolution and spatial analysis. Furthermore, we discuss the increasing application of deep learning, including foundation models, in analysing single-cell and spatial transcriptomics data. Finally, we discuss recent applications of these tools in the fields of stem cell biology, immunology, and tumour biology, and the future of single-cell and spatial transcriptomics in biological research and its translation to the clinic.

The hierarchy of needs for laboratory medicine requires a foundational care delivery model

Under the collective weight of growing test volume, staffing constraints, and Medicare reimbursements cuts, an enhancement-based, alternative payment structure focused on rewarding the laboratory's care delivery efforts via benchmarking is appealing. However, achieving a value-based payment model requires the development of an inclusive laboratory care delivery model (LCDM) framework. Today, a holistic, practical LCDM framework for laboratory medicine does not exist. However, such creation is essential for establishing unifying tenants of practice for value-tracing by which standardized key performance and population health indicators can be derived. LAB-CARES is the first step in formulating an LCDM with the primary objective of defining and streamlining the processes and strategies necessary to deliver and articulate the value of diagnostic excellence across the healthcare system. The goal of LAB-CARES is to maximize efficiencies, enhance quality, disseminate clinical expertise, increase patient safety, and promote integrative practice. LAB-CARES is designed to improve an individual patient's quality of life (longitudinal laboratory results – beyond one test) and their surrounding communities (e.g., through surveillance and prevention – beyond one patient). Further professional conversation and efforts are paramount to integrate LAB-CARES as a formalized structure within the healthcare landscape.

FENAP: Foundation Models for EMA-Derived Narrative Analysis and Prediction

FENAP: Foundation Models for EMA-Derived Narrative Analysis and Prediction

Laddering vision foundation model for remote sensing image change detection

Laddering vision foundation model for remote sensing image change detection

Quantized multi-task learning for context-specific representations of gene network dynamics.

While often represented as static entities, gene networks are highly context-dependent. Here, we developed a multi-task learning strategy to yield context-specific representations of gene network dynamics. We assembled a corpus comprising ~103 million human single-cell transcriptomes from a broad range of tissues and diseases and performed a two stage pretraining, first with non-malignant cells to generate a foundational model and then with continual learning on cancer cells to tune the model to the cancer domain. We performed multi-task learning with the foundational model to learn context-specific representations of a broad range of cell types, tissues, developmental stages, and diseases. We then leveraged the cancer-tuned model to jointly learn cell states and predict tumor-restricting factors within the colorectal tumor microenvironment. Model quantization allowed resource-efficient fine-tuning and inference while preserving biological knowledge. Overall, multi-task learning enables context-specific disease modeling that can yield contextual predictions of candidate therapeutic targets for human disease.

Dspace implementation of real-time selective harmonics elimination technique using modified carrier on three phase inverter

Introduction. In the contemporary world, alternative electrical energy has become an integral part of our daily existence, with the majority of our electrical materials, electronic devices, and industrial equipment relying on this energy source. Consequently, ensuring the quality of the electrical signal obtained is of paramount importance in the process of converting and distributing electrical energy. The improvement of the output voltage inverter can be achieved through adjustments to the inverter structure or by refining the control strategy. The novelty of the presented research lies in an innovative approach that employs real-time modulation for efficient control over the reduction of harmonics, alongside managing the fundamental component. This approach is applicable to both bipolar and unipolar configurations, featuring quarter-wave and half-wave symmetries. Purpose. Employing this modulation strategy aims to enhance the durability of the switching components and enhance the voltage output of the inverter. Methods. The methodology is founded on a sine-sine modulation as its foundational model. It constitutes an inventive pulse width modulation technique in which a reference sinusoidal waveform, operating at the desired signal frequency, is compared to a modified carrier signal with an identical time period as the reference signal. Results. This paper introduces a broader and more comprehensive approach, alongside specific solutions, for the control and mitigation of harmonics in three phase voltage source inverters. The proposed method offers precise control over the reduction of harmonics and the fundamental component, and it can be implemented in extensive power electronic converters. Practical value. To assess the effectiveness of the given control approach, we conducted simulations as well as real-time implementation employing Dspace DS1104 controller board. The outcomes were highly favorable, confirming the effectiveness and validity of the suggested control algorithm. References 15, tables 4, figures 7.

Scupa: Single-cell unified polarization assessment of immune cells using the single-cell foundation model.

Immune cells undergo cytokine-driven polarization in respond to diverse stimuli. This process significantly modulates their transcriptional profiles and functional states. Although single-cell RNA sequencing (scRNA-seq) has advanced our understanding of immune responses across various diseases or conditions, currently there lacks a method to systematically examine cytokine effects and immune cell polarization. To address this gap, we developed Single-cell unified polarization assessment (Scupa), the first computational method for comprehensive immune cell polarization analysis. Scupa is trained on data from the Immune Dictionary, which characterizes 66 cytokine-driven polarization states across 14 immune cell types. By leveraging the cell embeddings from the Universal Cell Embeddings model, Scupa effectively identifies polarized cells in new datasets generated from different species and experimental conditions. Applications of Scupa in independent datasets demonstrated its accuracy in classifying polarized cells and further revealed distinct polarization profiles in tumor-infiltrating myeloid cells across cancers. Scupa complements conventional single-cell data analysis by providing new insights into immune cell polarization, and it holds promise for assessing molecular effects or identifying therapeutic targets in cytokine-based therapies.