Foundation Model Research Articles

Validation of histopathology foundation models through whole slide image retrieval

We evaluated several foundation models in histopathology for image retrieval using a zero-shot approach. These models generated embeddings that were directly employed for retrieval without additional fine-tuning. Our experiments were conducted on diagnostic slides from The Cancer Genome Atlas (TCGA), which covers 23 organs and 117 cancer subtypes. We used Yottixel as the framework for whole-slide image (WSI) retrieval via patch-based embeddings. Retrieval performance was evaluated using macro-averaged F1 scores for top-1, top-3, and top-5 retrievals. The top-5 retrieval F1 scores indicated varying levels of performance: Yottixel-DenseNet (27% ± 13%), Yottixel-UNI (42% ± 14%), Yottixel-Virchow (40% ± 13%), Yottixel-GigaPath (41% ± 13%), and GigaPath WSI (40% ± 14%). These results demonstrate the potential and limitations of foundation models for histopathology image retrieval, underscoring the need for further advancements in embedding and retrieval techniques.

A foundation model with weak experiential guidance in detecting muscle invasive bladder cancer on MRI.

Preoperative detection of muscle-invasive bladder cancer (MIBC) remains a great challenge in practice. We aimed to develop and validate a deep Vesical Imaging Network (ViNet) model for the detection of MIBC using high-resolution T2-weighted MR imaging (hrT2WI) in a multicenter cohort. ViNet was designed using a modified 3D ResNet, in which, the encoder layers were pretrained using a self-supervised foundation model on over 40,000 cross-modal imaging datasets for transfer learning, and the classification modules were weakly supervised by an experiential knowledge-domain mask indicated by a nnUNet segmentation model. Optimal ViNet model was trained in derivation data (cohort 1, n=312) and validated in multicenter data (cohort 2, n=79; cohort 3, n=44; cohort 4, n=56) across a multi-ablation-test for model selection. In internal validation, ViNet using hrT2WI outperformed all ablation-test models (odds ratio [OR], 7.41 versus 1.85-2.70; all P<0.05). In external validation, the performance of ViNet using hrT2WI versus ablation-test models was heterogeneous (OR, 1.31-3.89 versus 0.89-9.75; P=0.03-0.15). In addition, clinical benefit of ViNet was evaluated between six readers using the Vesical Imaging-Reporting and Data System (VI-RADS) versus ViNet-adjusted VI-RADS. As a result, ViNet-adjusted VI-RADS upgraded 62.9% (17/27) of MIBC missed in VI-RADS score 2, while downgraded 84.1% (69/84), 62.5% (35/56) and 67.9% (19/28) of non-muscle-invasive bladder cancer (NMIBC) overestimated in VI-RADS score 3-5. We concluded that ViNet presents a promising alternative for diagnosing MIBC using hrT2WI instead of conventional multiparametric MRI.

A unified machine learning approach for reconstructing hadronically decaying tau leptons

Tau leptons serve as an important tool for studying the production of Higgs and electroweak bosons, both within and beyond the Standard Model of particle physics. Accurate reconstruction and identification of hadronically decaying tau leptons is a crucial task for current and future high energy physics experiments. Given the advances in jet tagging, we demonstrate how tau lepton reconstruction can be decomposed into tau identification, kinematic reconstruction, and decay mode classification in a multi-task machine learning setup. Based on an electron-positron collision dataset with full detector simulation and reconstruction, we show that common jet tagging architectures can be effectively used for these sub-tasks. We achieve comparable momentum resolutions of 2–3% with all the tested models, while the precision of reconstructing individual decay modes is between 80–95%. We find ParticleTransformer to be the best-performing approach, significantly outperforming the heuristic baseline. This paper also serves as an introduction to a new publicly available Fuτure dataset for the development of tau reconstruction algorithms. This allows to further study the resilience of ML models to domain shifts and the efficient use of foundation models for such tasks.

Learning the language of life with AI.

In 2021, a year before ChatGPT took the world by storm amid the excitement about generative artificial intelligence (AI), AlphaFold 2 cracked the 50-year-old protein-folding problem, predicting three-dimensional (3D) structures for more than 200 million proteins from their amino acid sequences. This accomplishment was a precursor to an unprecedented burgeoning of large language models (LLMs) in the life sciences. That was just the beginning. In recent months, we have moved into a hyperaccelerated phase of new foundation models, pretrained on massive datasets, with the ability to perform a wide range of tasks that are helping us understand the structure, biology, evolution, and design of proteins, RNA, DNA, and ligands, as well as their biomolecular interactions. Unlike multimodal LLMs such as GPT-4, Gemini, and Claude, which process text, audio, and images, these large language of life models (LLLMs) are multiomic. That is to say, they are not only multimodal but pertain to different layers of molecular biology. For example, Evo, a foundation model trained on 2.7 million diverse phage and prokaryotic genomes (equivalent to about 300 billion DNA nucleotides), predicts the impact of variants in DNA, RNA, or proteins on structure and function, as well as how essential genes are to cell function, and can generate new DNA sequences.

A comprehensive evaluation of histopathology foundation models for ovarian cancer subtype classification

Histopathology foundation models show great promise across many tasks, but analyses have been limited by arbitrary hyperparameters. We report the most rigorous single-task validation study to date, specifically in the context of ovarian carcinoma morphological subtyping. Attention-based multiple instance learning classifiers were compared using three ImageNet-pretrained encoders and fourteen foundation models, each trained with 1864 whole slide images and validated through hold-out testing and two external validations (the Transcanadian Study and OCEAN Challenge). The best-performing classifier used the H-optimus-0 foundation model, with balanced accuracies of 89%, 97%, and 74%, though UNI achieved similar results at a quarter of the computational cost. Hyperparameter tuning the classifiers improved performance by a median 1.9% balanced accuracy, with many improvements being statistically significant. Foundation models improve classification performance and may allow for clinical utility, with models providing a second opinion in challenging cases and potentially improving the accuracy and efficiency of diagnoses.

From Reactions to Reflection: A Recursive Framework for the Evolution of Cognition and Complexity.

This paper presents a comprehensive framework that traces the evolution of consciousness through a continuum of recursive processes spanning reaction, temporogenesis, symbiogenesis, and cognogenesis. By integrating biological cooperation, temporal structuring, and self-referential processing, our model provides a novel perspective on how complexity emerges and scales across evolutionary time. Reaction is established as the foundational mechanism that enables adaptive responses to environmental stimuli, which, through recursive refinement, transitions into temporogenesis-the synchronization of internal processes with external temporal rhythms. Symbiogenesis further enhances this process by fostering cooperative interactions at multiple biological levels, facilitating the emergence of higher-order cognitive functions. Cognogenesis represents the culmination of these recursive processes, where self-awareness and intentionality arise through iterative feedback loops. Our framework offers a biologically grounded pathway to addressing the "hard problem" of consciousness by proposing that subjective experience emerges as a result of progressively complex recursive interactions rather than as a static or isolated phenomenon. In comparing our approach with established theories such as Integrated Information Theory, Global Workspace Theory, and enactive cognition, we highlight its unique contributions in situating consciousness within a broader evolutionary and biological context. This work aims to provide a foundational model that bridges the gap between reaction and reflection, offering empirical avenues for further exploration in neuroscience, evolutionary biology, and artificial intelligence.

Causality-driven feature selection and domain adaptation for enhancing chemical foundation models in downstream tasks

Abstract Recent advancements in large foundation models have revealed impressive capabilities in mastering complex chemical language representations. These models undergo a task-agnostic learning phase, characterized by pre-training on extensive unlabeled corpora followed by fine-tuning on specific downstream tasks. This methodology reduces reliance on labeled data, facilitating data acquisition and broadening the scope of chemical language representation. However, real-world scenarios often pose challenges due to domain shift, a phenomenon where the data distribution in downstream tasks differs from that of the pre-training phase, potentially degrading model performance. To address this, we present a novel causal-based framework for feature selection and domain adaptation to enhance the performance of chemical foundation models on downstream tasks. Our approach employs a multi-stage feature selection method that identifies physico-chemical features based on their direct causal-effect over specific downstream properties. By employing Mordred descriptors and Markov blanket causal graphs, our approach provides insight into the causal relationships between features and target properties for prediction tasks. We evaluate our approach on various foundation model architectures and datasets, demonstrating performance improvements, which showcases the robustness and the agnostic nature of our approach.

Foundation Models for Translational Cancer Biology.

Cancer remains a leading cause of death globally. The complexity and diversity of cancer-related datasets across different specialties pose challenges in refining precision medicine for oncology. Foundation models offer a promising solution. Trained on vast amounts of data, these models develop a broad understanding across a wide range of tasks. We examine the role of foundation models in domains relevant to cancer research, including natural language processing, computer vision, molecular biology, and cheminformatics. Through a review of state-of-the-art methods, we explore how these models have already advanced translational cancer research goals such as precision tumor classification and artificial intelligence-assisted surgery. We also discuss prospective advances in areas like early tumor detection, personalized cancer treatment, and drug discovery. This review provides researchers with a curated set of resources and methodologies, offers practitioners a deeper understanding of how these models enhance cancer care, and points to opportunities for future applications of foundation models in cancer research.

Digital framework for georeferenced multiplatform surveillance of banana wilt using human in the loop AI and YOLO foundation models

Bananas (Musa spp.) are a critical global food crop, providing a primary source of nutrition for millions of people. Traditional methods for disease monitoring and detection are often time-consuming, labor-intensive, and prone to inaccuracies. This study introduces an AI-powered multiplatform georeferenced surveillance system designed to enhance the detection and management of banana wilt diseases. We developed and evaluated several deep learning foundation models, including YOLO-NAS, YOLOv8, YOLOv9, and Faster-RCNN to perform accurate disease detection on both platforms. Our results demonstrate the superior performance of YOLOv9 in detecting healthy, Fusarium Wilt and Xanthomonas Wilt diseased plants in aerial images, achieving high mAP@50, precision and recall metrics ranging from 55 to 86%. In terms of ground level images, we organized the dataset based on disease occurrence in Africa, Latin America, India, Asia and Australia. For this platform, YOLOv8 outperforms the rest and achieves mAP@50, precision and recall between 65 and 99% depending on the plant part and region. Additionally, we incorporated Explainable AI techniques, such as Gradient-weighted Class Activation Mapping, to enhance model transparency and trustworthiness. Human in the Loop Artificial Intelligence was also utilized to enhance the ground level model’s predictions.

Beyond scaleup: Knowledge‐aware parsimony learning from deep networks

AbstractThe brute‐force scaleup of training datasets, learnable parameters and computation power, has become a prevalent strategy for developing more robust learning models. However, due to bottlenecks in data, computation, and trust, the sustainability of this strategy is a serious concern. In this paper, we attempt to address this issue in a parsimonious manner (i.e., achieving greater potential with simpler models). The key is to drive models using domain‐specific knowledge, such as symbols, logic, and formulas, instead of purely relying on scaleup. This approach allows us to build a framework that uses this knowledge as “building blocks” to achieve parsimony in model design, training, and interpretation. Empirical results show that our methods surpass those that typically follow the scaling law. We also demonstrate our framework in AI for science, specifically in the problem of drug‐drug interaction prediction. We hope our research can foster more diverse technical roadmaps in the era of foundation models.

Multimodal learning for mapping genotype-phenotype dynamics.

How complex phenotypes emerge from intricate gene expression patterns is a fundamental question in biology. Integrating high-content genotyping approaches such as single-cell RNA sequencing and advanced learning methods such as language models offers an opportunity for dissecting this complex relationship. Here we present a computational integrated genetics framework designed to analyze and interpret the high-dimensional landscape of genotypes and their associated phenotypes simultaneously. We applied this approach to develop a multimodal foundation model to explore the genotype-phenotype relationship manifold for human transcriptomics at the cellular level. Analyzing this joint manifold showed a refined resolution of cellular heterogeneity, uncovered potential cross-tissue biomarkers and provided contextualized embeddings to investigate the polyfunctionality of genes shown for the von Willebrand factor (VWF) gene in endothelial cells. Overall, this study advances our understanding of the dynamic interplay between gene expression and phenotypic manifestation and demonstrates the potential of integrated genetics in uncovering new dimensions of cellular function and complexity.

Evolutionary optimization of model merging recipes

Abstract Large language models (LLMs) have become increasingly capable, but their development often requires substantial computational resources. Although model merging has emerged as a cost-effective promising approach for creating new models by combining existing ones, it currently relies on human intuition and domain knowledge, limiting its potential. Here we propose an evolutionary approach that overcomes this limitation by automatically discovering effective combinations of diverse open-source models, harnessing their collective intelligence without requiring extensive additional training data or compute. Our approach operates in both parameter space and data flow space, allowing optimization beyond just the weights of the individual models. This approach even facilitates cross-domain merging, generating models such as a Japanese LLM with math reasoning capabilities. Surprisingly, our Japanese math LLM achieved state-of-the-art performance on a variety of established Japanese LLM benchmarks, even surpassing models with substantially more parameters, despite not being explicitly trained for such tasks. Furthermore, a culturally aware Japanese vision–language model generated through our approach demonstrates its effectiveness in describing Japanese culture-specific content, outperforming previous Japanese vision–language models. This work not only contributes new state-of-the-art models back to the open-source community but also introduces a new paradigm for automated model composition, paving the way for exploring alternative, efficient approaches to foundation model development.

Full fine-tuning strategy for endoscopic foundation models with expanded learnable offset parameters.

In the medical field, endoscopic video analysis is crucial for disease&#xD;diagnosis and minimally invasive surgery. The Endoscopic Foundation Models (Endo-&#xD;FM) utilize large-scale self-supervised pre-training on endoscopic video data and&#xD;leverage video transformer models to capture long-range spatiotemporal dependencies.&#xD;However, detecting complex lesions such as gastrointestinal metaplasia (GIM) in&#xD;endoscopic videos remains challenging due to unclear boundaries and indistinct&#xD;features, and Endo-FM has not demonstrated good performance. To this end, we&#xD;propose a fully fine-tuning strategy with an Extended Learnable Offset Parameter&#xD;(ELOP), which improves model performance by introducing learnable offset parameters&#xD;in the input space. Specifically, we propose a novel loss function that combines cross-&#xD;entropy loss and focal loss through a weighted sum, enabling the model to better focus&#xD;on hard-to-classify samples during training. We validated ELOP on a private GIM&#xD;dataset from a local grade-A tertiary hospital and a public polyp detection dataset.&#xD;Experimental results show that ELOP significantly improves the detection accuracy,&#xD;achieving accuracy improvements of 6.25 % and 3.75%respectively compared to the&#xD;original Endo-FM. In summary, ELOP provides an excellent solution for detecting&#xD;complex lesions in endoscopic videos, achieving more precise diagnoses.

Current–Pressure Dynamics Modeling on an Annular Magnetorheological Valve for an Adaptive Rehabilitation Device for Disabled Individuals

The dynamic relationship between current and pressure in magnetorheological (MR) valves is essential for the design of adaptive rehabilitation devices aimed at health rehabilitation for disabled individuals, yet it remains under-explored in existing modeling approaches. Accurately capturing this relationship is vital to predict the pressure drop response to current variations, facilitating the development of effective control systems in such rehabilitation applications. This study employs a linear black-box modeling approach to characterize the current–pressure dynamics of an annular MR valve. Experimental data are used to develop a set of transfer function models, with parameters identified through MATLAB’s system identification tools, utilizing invariant variable regression and the Levenberg–Marquardt (LM) iteration. The modeling yielded a 14th-order transfer function, labeled TF14, which closely aligns with experimental data, achieving a root mean square error of 12.64%. These findings contribute valuable insights into the current–pressure dynamics of MR valves and establish a foundational model for adaptive rehabilitation devices designed for individuals with disabilities.

Abstract IA05: Toward multi-modal foundation AI for precision oncology

Abstract Clinical decision-making is a complex process that involves information obtained from multiple data modalities. Artificial intelligence (AI) approaches that can effectively integrate multi-modal data hold significant promise to advance clinical care. Two areas of success are imaging and digital pathology, where AI has shown great potential to improve cancer diagnosis and treatment. This talk will provide an overview on how AI can be used to extract information from imaging and pathology and identify prognostic and predictive biomarkers for personalized cancer treatment. In particular, I will present our recent work on building multi-modal foundation models for precision oncology. Citation Format: Ruijiang Li. Toward multi-modal foundation AI for precision oncology. [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Translating Targeted Therapies in Combination with Radiotherapy; 2025 Jan 26-29; San Diego, CA. Philadelphia (PA): AACR; Clin Cancer Res 2025;31(2_Suppl):Abstract nr IA05

Foundation models in bioinformatics

Abstract With the adoption of Foundation Models (FMs), Artificial Intelligence (AI) has become increasingly significant in bioinformatics and has successfully addressed many historical challenges, such as pre-training frameworks, model evaluation, and interpretability. FMs demonstrate notable proficiency in managing large-scale, unlabeled datasets, because experimental procedures are costly and labor-intensive. In various downstream tasks, FMs have consistently achieved noteworthy results, demonstrating high levels of accuracy in representing biological entities. A new era in computational biology has been ushered in by the application of FMs, focusing on both general and specific biological issues. In this review, we introduce recent advancements in bioinformatics FMs that employed in a variety of downstream tasks, including genomics, transcriptomics, proteomics, drug discovery, and single cell analysis. Our aim is to assist scientists in selecting appropriate FMs in bioinformatics, according to four model types: language FMs, vision FMs, graph FMs, and multimodal FMs. In addition to understanding molecular landscapes, AI technology can establish the theoretical and practical foundation for continued innovation in molecular biology.

Covid-19 Pandemic as a Natural Experiment: The Case of Home Advantage in Sports

Establishing a causal relationship requires not only the presence of a factor of interest but also the demonstration that the relationship is absent when the factor is absent. Such ideal conditions are rare, especially in observational studies in which creating control conditions is inherently difficult. The COVID-19 pandemic, with its unparalleled disruptions, offers a unique opportunity to isolate causal effects and test previously impossible hypotheses. Here, we examine the home advantage (HA) in sports—a phenomenon in which teams generally perform better in front of their home fans—and how the pandemic-induced absence of fans offered a fortunate yet systematic change to typical conditions, serving as a natural experiment. Using a structural equation modeling approach and building a mediation model encompassing all relevant HA factors, we quantified the reduction in HA and elucidated the specific mechanisms behind it. The theory behind HA and the availability of measures for each factor before and during COVID-19 lockdowns enabled us to estimate all postulated pathways within a natural experimental context. The robust statistical framework used in our study offers a foundational model for integrating naturally occurring events that serve as control conditions into the analysis of various real-life phenomena.

Large AI Models and Their Applications: Classification, Limitations, and Potential Solutions

ABSTRACTBackgroundIn recent years, Large Models (LMs) have been rapidly developed, including large language models, visual foundation models, and multimodal LMs. They are updated and iterated at a very fast pace. These LMs can accomplish many tasks, e.g., daily work assistant, intelligent customer service, and intelligent factory scheduling. Their development has contributed to various industries in human society.AimsThe architectural flaws of LMs lead to several problems, including illusions and difficulty in locating errors, limiting their performance. Solving these problems properly can facilitate their further development.MethodsThis work first introduces the development of LMs and identifies their current problems, including data and energy consumption, catastrophic forgetting, reasoning ability, localization fault, and ethical problems. Then, potential solutions to these problems are provided, including increase data and computation capability, neural‐symbolic synergy, and data orientation to human pattern.DiscussionThis work discusses developing vertical domain LMs on top of some base LMs. In addition, this work introduces three typical real‐world applications of LMs, including autonomous driving, smart industrial productions, and intelligent medical assistance.ConclusionBy embracing the advantages of LMs and solving their fundamental problems, many industries are expected to achieve promising prospects in the future.

P0340 Endo-Histo Foundational Fusion Model: A Novel Artificial Intelligence Approach for Predicting Histologic Remission and Early Response to Therapy in a Phase 2 Ulcerative Colitis Clinical Trial

Abstract Background Artificial Intelligence (AI)- enabled endoscopy and histology offer accurate, objective, and rapid assessment of disease activity in Ulcerative Colitis (UC). Emerging multi-source AI models integrating diverse datasets may enhance standardised disease evaluation and outcome prediction. This study aimed to develop a novel AI model fusing endoscopic and histological findings to improve the assessment of disease remission and predict early response to therapy in UC clinical trials. Methods A novel multimodal AI fusion algorithm was developed by integrating paired endoscopic videos and histological whole-slide images (WSIs) from the phase 2 clinical trial of Mirikizumab in UC (NCT02589665). The endoscopy branch of the model was trained with 291 white-light videos, using a convolutional neural network to select informative frames and the BioMedCLIP foundational model to extract features. The histology branch utilised the CONCH foundational model and was fine-tuned on 291 WSIs to obtain patch-level features. Features from both modalities were aggregated and fused using multi-head self-attention. The model’s ability to assess histological remission and response to therapy at weeks 12 and 52 was evaluated. Histological remission was defined as Geboes ≤2B.0, while a response to therapy at different time points was based on histological remission or improvement (Geboes &amp;lt;3.1). Results The fusion model outperformed single-modality assessments for histological remission, achieving a sensitivity of 89.72% (95% CI: 82.35–94.76), specificity of 89.67% (95% CI: 84.34–93.67), and accuracy of 89.69% (95% CI: 85.61–92.94). It demonstrated remarkable performance in assessing response to therapy at 12 and 52 weeks, with sensitivity of 97.96% (95% CI: 89.15–99.95), specificity of 86.84% (95% CI: 71.91–95.59) and accuracy of 93.10% (95% CI 85.59 – 97.43) for histological remission at week 52. Substantial agreement was observed between the AI fusion model and central readout. Conclusion This innovative multimodal fusion AI model enhances the assessment of histological remission and accurately predicts response to therapy. By potentially standardising central readouts and enabling automated disease assessment, this novel tool marks a significant advancement towards precision medicine in clinical trials.

The Voltage Oracle: A Foundation Model for Probabilistic PEM Fuel Cell Voltage Prediction

Abstract Long-term operation of proton exchange membrane (PEM) fuel cells requires precise monitoring to detect and prevent harmful anomalies. Cell voltage is a sensitive indicator for the fuel cell's state of health. However, its dependency on fuel cell operating conditions impedes the detection of anomalies. To overcome this limitation, we present a machine learning driven foundation model for probabilistic fuel cell voltage prediction. The foundation model can be fitted to new fuel cell designs using transfer learning for anomaly detection.&amp;#xD;Since the availability and variance in real fuel cell test data sets is very limited, a physics-based fuel cell simulation model is used to generate 30,000 data points to train the machine learning model. To improve the prediction accuracy of the foundation model, additional steps such as feature engineering and target balancing are introduced and their individual effects are analyzed. The performance of the pre-trained foundation model is then tested by transfer learning the foundation model using real measurement data from two different automotive fuel cell designs. It is shown that with sufficient training data, the machine learning model outperforms the physics-based simulation model by an RMSE improvement of up to 37% and respectively by 61% on the test data sets.