Foundation Model Research Articles

Artificial intelligence in surgery.

Artificial intelligence (AI) is rapidly emerging in healthcare, yet applications in surgery remain relatively nascent. Here we review the integration of AI in the field of surgery, centering our discussion on multifaceted improvements in surgical care in the preoperative, intraoperative and postoperative space. The emergence of foundation model architectures, wearable technologies and improving surgical data infrastructures is enabling rapid advances in AI interventions and utility. We discuss how maturing AI methods hold the potential to improve patient outcomes, facilitate surgical education and optimize surgical care. We review the current applications of deep learning approaches and outline a vision for future advances through multimodal foundation models.

Prospective Role of Foundation Models in Advancing Autonomous Vehicles

Prospective Role of Foundation Models in Advancing Autonomous Vehicles

Fine-tuning vision foundation model for crack segmentation in civil infrastructures

Large-scale foundation models have become the mainstream deep learning method, while in civil engineering, the scale of AI models is strictly limited. In this work, a vision foundation model is introduced for crack segmentation. Two parameter-efficient fine-tuning methods, adapter and low-rank adaptation, are adopted to fine-tune the foundation model in semantic segmentation: the Segment Anything Model (SAM). The fine-tuned CrackSAM shows excellent performance on different scenes and materials. To test the zero-shot performance of the proposed method, two unique datasets related to road and exterior wall cracks are collected, annotated and open-sourced, for a total of 810 images. Comparative experiments are conducted with twelve mature semantic segmentation models. On datasets with artificial noise and previously unseen datasets, the performance of CrackSAM far exceeds that of all state-of-the-art models. CrackSAM exhibits remarkable superiority, particularly under challenging conditions such as dim lighting, shadows, road markings, construction joints, and other interference factors. These cross-scenario results demonstrate the outstanding zero-shot capability of foundation models and provide new ideas for developing vision models in civil engineering.

A Transformer and Visual Foundation Model-Based Method for Cross-View Remote Sensing Image Retrieval

Abstract. Retrieving UAV images that lack POS information with georeferenced satellite orthoimagery is challenging due to the differences in angles of views. Most existing methods rely on deep neural networks with a large number of parameters, leading to substantial time and financial investments in network training. Consequently, these methods may not be well-suited for downstream tasks that have high timeliness requirements. In this work, we propose a cross-view remote sensing image retrieval method based on transformer and visual foundation model. We investigated the potential of visual foundation model for extracting common features from cross-view images. Training is only conducted on a small, self-designed retrieval head, alleviating the burden of network training. Specifically, we designed a CVV module to optimize the features extracted from the visual foundation model, making these features more adept for cross-view image retrieval tasks. And we designed an MLP head to achieve similarity discrimination. The method is verified on a publicly available dataset containing multiple scenes. Our method shows excellent results in terms of both efficiency and accuracy on 15 sub-datasets (10 or 50 scene categories) derived from the public dataset, which holds practical value in engineering applications with streamlined scene categories and constrained computational resources. Furthermore, we initiated a comprehensive discussion and conducted ablation experiments on the network design to validate its efficacy. Additionally, we analyzed the presence of overfitting within the network and deliberated on the limitations of our study, proposing potential avenues for future enhancements.

A Novel Pretrained General-purpose Vision Language Model for the Vietnamese Language

Lying in the cross-section of computer vision and natural language processing, vision language models are capable of processing images and text at once. These models are helpful in various tasks: text generation from image and vice versa, image-text retrieval, or visual navigation. Besides building a model trained on a dataset for a task, people also study general-purpose models to utilize many datasets for multitasks. Their two primary applications are image captioning and visual question answering. For English, large datasets and foundation models are already abundant. However, for Vietnamese, they are still limited. To expand the language range, this work proposes a pretrained general-purpose image-text model named VisualRoBERTa. A dataset of 600k images with captions (translated MS COCO 2017 from English to Vietnamese) is introduced to pretrain VisualRoBERTa. The model’s architecture is built using Convolutional Neural Network and Transformer blocks. Fine-tuning VisualRoBERTa shows promising results on the ViVQA dataset with 34.49% accuracy, 0.4173 BLEU 4, and 0.4390 RougeL (in visual question answering task), and best outcomes on the sViIC dataset with 0.6685 BLEU 4, 0.6320 RougeL (in image captioning task).

Generative AI and process systems engineering: The next frontier

This review article explores how emerging generative artificial intelligence (GenAI) models, such as large language models (LLMs), can enhance solution methodologies within process systems engineering (PSE). These cutting-edge GenAI models, particularly foundation models (FMs), which are pre-trained on extensive, general-purpose datasets, offer versatile adaptability for a broad range of tasks, including responding to queries, image generation, and complex decision-making. Given the close relationship between advancements in PSE and developments in computing and systems technologies, exploring the synergy between GenAI and PSE is essential. We begin our discussion with a compact overview of both classic and emerging GenAI models, including FMs, and then dive into their applications within key PSE domains: synthesis and design, optimization and integration, and process monitoring and control. In each domain, we explore how GenAI models could potentially advance PSE methodologies, providing insights and prospects for each area. Furthermore, the article identifies and discusses potential challenges in fully leveraging GenAI within PSE, including multiscale modeling, data requirements, evaluation metrics and benchmarks, and trust and safety, thereby deepening the discourse on effective GenAI integration into systems analysis, design, optimization, operations, monitoring, and control. This paper provides a guide for future research focused on the applications of emerging GenAI in PSE.

Deep Learning-Enabled Quantification of 99mTc-Pyrophosphate SPECT/CT for Cardiac Amyloidosis.

Transthyretin cardiac amyloidosis (ATTR CA) is increasingly recognized as a cause of heart failure in older patients, with 99mTc-pyrophosphate imaging frequently used to establish the diagnosis. Visual interpretation of SPECT images is the gold standard for interpretation but is inherently subjective. Manual quantitation of SPECT myocardial 99mTc-pyrophosphate activity is time-consuming and not performed clinically. We evaluated a deep learning approach for fully automated volumetric quantitation of 99mTc-pyrophosphate using segmentation of coregistered anatomic structures from CT attenuation maps. Methods: Patients who underwent SPECT/CT 99mTc-pyrophosphate imaging for suspected ATTR CA were included. Diagnosis of ATTR CA was determined using standard criteria. Cardiac chambers and myocardium were segmented from CT attenuation maps using a foundational deep learning model and then applied to attenuation-corrected SPECT images to quantify radiotracer activity. We evaluated the diagnostic accuracy of target-to-background ratio (TBR), cardiac pyrophosphate activity (CPA), and volume of involvement (VOI) using the area under the receiver operating characteristic curve (AUC). We then evaluated associations with the composite outcome of cardiovascular death or heart failure hospitalization. Results: In total, 299 patients were included (median age, 76 y), with ATTR CA diagnosed in 83 (27.8%) patients. CPA (AUC, 0.989; 95% CI, 0.974-1.00) and VOI (AUC, 0.988; 95% CI, 0.973-1.00) had the highest prediction performance for ATTR CA. The next highest AUC was for TBR (AUC, 0.979; 95% CI, 0.964-0.995). The AUC for CPA was significantly higher than that for heart-to-contralateral ratio (AUC, 0.975; 95% CI, 0.952-0.998; P = 0.046). Twenty-three patients with ATTR CA experienced cardiovascular death or heart failure hospitalization. All methods for establishing TBR, CPA, and VOI were associated with an increased risk of events after adjustment for age, with hazard ratios ranging from 1.41 to 1.84 per SD increase. Conclusion: Deep learning segmentation of coregistered CT attenuation maps is not affected by the pattern of radiotracer uptake and allows for fully automatic quantification of hot-spot SPECT imaging such as 99mTc-pyrophosphate. This approach can be used to accurately identify patients with ATTR CA and may play a role in risk prediction.

Theoretical Solutions for Forecasting the Response of the Existing Pipeline Induce by Tunneling underneath

In order to accurately and efficiently assess the impact of tunnel excavation on overlying existing pipeline, an analytical method is proposed to solve this problem. First, the vertical free displacement of the surrounding soil due to tunnel excavation can be derived by the Loganathan formula. Next, the overlying existing pipeline can be treated as a Timoshenko beam resting on the Vlasov foundation model, and the influence of the surrounding soil on the both sides of the existing pipeline is taken into consideration. Finally, an analytical solution for the longitudinal deformation of the existing pipeline can be obtained by using the integral method. Case analysis results demonstrate that the calculated results of this method closely in line with measured data. Compared to the degenerate analytical solution given by this method, the result from this method is more consistent with the measured data. Further parameter studies show that the volume loss rate, diameter of new tunnel, skew angle, and vertical distance between tunnel and pipeline are significant factors affecting the existing pipeline response due to tunneling underlying.

The new paradigm in machine learning - foundation models, large language models and beyond: a primer for physicians.

Foundation machine learning models are deep learning models capable of performing many different tasks using different data modalities such as text, audio, images and video. They represent a major shift from traditional task-specific machine learning prediction models. Large language models (LLM), brought to wide public prominence in the form of ChatGPT, are text-based foundational models that have the potential to transform medicine by enabling automation of a range of tasks, including writing discharge summaries, answering patients questions and assisting in clinical decision-making. However, such models are not without risk and can potentially cause harm if their development, evaluation and use are devoid of proper scrutiny. This narrative review describes the different types of LLM, their emerging applications and potential limitations and bias and likely future translation into clinical practice.

ATTACK-COSM: attacking the camouflaged object segmentation model through digital world adversarial examples

AbstractThe camouflaged object segmentation model (COSM) has recently gained substantial attention due to its remarkable ability to detect camouflaged objects. Nevertheless, deep vision models are widely acknowledged to be susceptible to adversarial examples, which can mislead models, causing them to make incorrect predictions through imperceptible perturbations. The vulnerability to adversarial attacks raises significant concerns when deploying COSM in security-sensitive applications. Consequently, it is crucial to determine whether the foundational vision model COSM is also susceptible to such attacks. To our knowledge, our work represents the first exploration of strategies for targeting COSM with adversarial examples in the digital world. With the primary objective of reversing the predictions for both masked objects and backgrounds, we explore the adversarial robustness of COSM in full white-box and black-box settings. In addition to the primary objective of reversing the predictions for masked objects and backgrounds, our investigation reveals the potential to generate any desired mask through adversarial attacks. The experimental results indicate that COSM demonstrates weak robustness, rendering it vulnerable to adversarial example attacks. In the realm of COS, the projected gradient descent (PGD) attack method exhibits superior attack capabilities compared to the fast gradient sign (FGSM) method in both white-box and black-box settings. These findings reduce the security risks in the application of COSM and pave the way for multiple applications of COSM.

An open-source research platform for mechanical ventilation based on Simulink® and STM32 nucleo

Abstract Gain in knowledge in experimental fields of research is mostly achieved using commercial hardware platforms. The linked costs however can limit a broad access by scientists and in the specific case of mechanical ventilation the pandemic demonstrated that further and collective research is essential and necessary. This contribution presents the modular design and implementation of a research platform which is based on STM32 Nucleo boards and Simulink®. The hardware required such as the pneumatic circuit are covered as well as the software architecture in which clinician related settings are processed for control references. The core functionality of mandatory volume controlled (VCV) and pressure controlled ventilation (PCV) while ensuring a positive end-expiratory pressure for the patient ( PEEP ) $(\text{PEEP})$ is evaluated and the foundational software models are published in a repository.

HydrogelFinder: A Foundation Model for Efficient Self-Assembling Peptide Discovery Guided by Non-Peptidal Small Molecules.

Self-assembling peptides have numerous applications in medicine, food chemistry, and nanotechnology. However, their discovery has traditionally been serendipitous rather than driven by rational design. Here, HydrogelFinder, a foundation model is developed for the rational design of self-assembling peptides from scratch. This model explores the self-assembly properties by molecular structure, leveraging 1,377 self-assembling non-peptidal small molecules to navigate chemical space and improve structural diversity. Utilizing HydrogelFinder, 111 peptide candidates are generated and synthesized 17 peptides, subsequently experimentally validating the self-assembly and biophysical characteristics of nine peptides ranging from 1-10 amino acids-all achieved within a 19-day workflow. Notably, the two de novo-designed self-assembling peptides demonstrated low cytotoxicity and biocompatibility, as confirmed by live/dead assays. This work highlights the capacity of HydrogelFinder to diversify the design of self-assembling peptides through non-peptidal small molecules, offering a powerful toolkit and paradigm for future peptide discovery endeavors.

Exploration of an Open Vocabulary Model on Semantic Segmentation for Street Scene Imagery

This study investigates the efficacy of an open vocabulary, multi-modal, foundation model for the semantic segmentation of images from complex urban street scenes. Unlike traditional models reliant on predefined category sets, Grounded SAM uses arbitrary textual inputs for category definition, offering enhanced flexibility and adaptability. The model’s performance was evaluated across single and multiple category tasks using the benchmark datasets Cityscapes, BDD100K, GTA5, and KITTI. The study focused on the impact of textual input refinement and the challenges of classifying visually similar categories. Results indicate strong performance in single-category segmentation but highlighted difficulties in multi-category scenarios, particularly with categories bearing close textual or visual resemblances. Adjustments in textual prompts significantly improved detection accuracy, though challenges persisted in distinguishing between visually similar objects such as buses and trains. Comparative analysis with state-of-the-art models revealed Grounded SAM’s competitive performance, particularly notable given its direct inference capability without extensive dataset-specific training. This feature is advantageous for resource-limited applications. The study concludes that while open vocabulary models such as Grounded SAM mark a significant advancement in semantic segmentation, further improvements in integrating image and text processing are essential for better performance in complex scenarios.

Artificial intelligence-based differential diagnosis of orbital MALT lymphoma and IgG4 related ophthalmic disease using hematoxylin-eosin images.

To investigate the possibility of distinguishing between IgG4-related ophthalmic disease (IgG4-ROD) and orbital MALT lymphoma using artificial intelligence (AI) and hematoxylin-eosin (HE) images. After identifying a total of 127 patients from whom we were able to procure tissue blocks with IgG4-ROD and orbital MALT lymphoma, we performed histological and molecular genetic analyses, such as gene rearrangement. Subsequently, pathological HE images were collected from these patients followed by the cutting out of 10 different image patches from the HE image of each patient. A total of 970 image patches from the 97 patients were used to construct nine different models of deep learning, and the 300 image patches from the remaining 30 patients were used to evaluate the diagnostic performance of the models. Area under the curve (AUC) and accuracy (ACC) were used for the performance evaluation of the deep learning models. In addition, four ophthalmologists performed the binary classification between IgG4-ROD and orbital MALT lymphoma. EVA, which is a vision-centric foundation model to explore the limits of visual representation, was the best deep learning model among the nine models. The results of EVA were ACC = 73.3% and AUC = 0.807. The ACC of the four ophthalmologists ranged from 40 to 60%. It was possible to construct an AI software based on deep learning that was able to distinguish between IgG4-ROD and orbital MALT. This AI model may be useful as an initial screening tool to direct further ancillary investigations.

Foundation Models, Generative AI, and Large Language Models

We are in a booming era of artificial intelligence, particularly with the increased availability of technologies that can help generate content, such as ChatGPT. Healthcare institutions are discussing or have started utilizing these innovative technologies within their workflow. Major electronic health record vendors have begun to leverage large language models to process and analyze vast amounts of clinical natural language text, performing a wide range of tasks in healthcare settings to help alleviate clinicians' burden. Although such technologies can be helpful in applications such as patient education, drafting responses to patient questions and emails, medical record summarization, and medical research facilitation, there are concerns about the tools' readiness for use within the healthcare domain and acceptance by the current workforce. The goal of this article is to provide nurses with an understanding of the currently available foundation models and artificial intelligence tools, enabling them to evaluate the need for such tools and assess how they can impact current clinical practice. This will help nurses efficiently assess, implement, and evaluate these tools to ensure these technologies are ethically and effectively integrated into healthcare systems, while also rigorously monitoring their performance and impact on patient care.

Interfacial Stress and Crack Propagation Experimental Study in Mini-LED Chip Debonding

The ultra-miniaturization trend of chips presents new challenges for non-destructive peeling-off techniques. This study models the mini-LED chip-UV film adhesive system as a micro laminated structure and conducts the interfacial stresses and crack propagation analysis of the structure using theoretical, finite element and experimental methods. The innovations of this paper are the illustration of the negligible effect of size effect on 125 μm x 75 μm x 60 μm chips, and the determination of the technology parameter tuning method to promote chip peeling-off. Firstly, applying the first-order shear deformation theory, and considering the UV-curable acrylic adhesive layer and the UV film as elastic materials, Lamé coefficients are introduced to explore the free vibration of the micro laminated structure. The analyses indicate that the scale effect has an extremely small impact, within 0.5%, on natural frequencies of the 100-micron scale laminated structure with a small length-to-thickness ratio. Thereafter, a three parameter elastic foundation (3-PEF) model of the chip-UV film adhesive structure is established and the interfacial stresses distributions are parametrically analyzed. Additionally, an experimental platform for needle-ejecting chip peeling-off is designed and constructed. Finite element simulation, experimental observations, and image recognition of crack generation and propagation during the peeling process were performed. Results show that cracks are generated at both free ends of the adhesive layer and progressively extended inwards. This is consistent with the 3-PEF analytical solution, in which the stress concentration also occurs at free ends of the adhesive layer. The agreement between theoretical, simulation and experimental results regarding interfacial stresses provides a solid foundation for further analysis of chip peeling-off mechanism.

Matte anything: Interactive natural image matting with segment anything model

Natural image matting algorithms aim to predict the transparency map (alpha-matte) with the trimap guidance. However, the production of trimap often requires significant labor, which limits the widespread application of matting algorithms on a large scale. To address the issue, we propose Matte Anything model (MatAny), an interactive natural image matting model that could produce high-quality alpha-matte with various simple hints. The key insight of MatAny is to generate pseudo trimap automatically with contour and transparency prediction. In our work, we leverage vision foundation models to enhance the performance of natural image matting. Specifically, we use the segment anything model to predict high-quality contour with user interaction and an open-vocabulary detector to predict the transparency of any object. Subsequently, a pre-trained image matting model generates alpha mattes with pseudo trimaps. MatAny is the interactive matting algorithm with the most supported interaction methods and the best performance to date. It consists of orthogonal vision models without any additional training. We evaluate the performance of MatAny against several previous image matting algorithms. MatAny has 58.3% improvement on MSE and 40.6% improvement on SAD compared to the previous image matting methods with simple guidance, achieving new state-of-the-art (SOTA) performance. The source codes and pre-trained models are available at https://github.com/hustvl/Matte-Anything.

PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models – Federated Learning in Age of Foundation Model

PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models – Federated Learning in Age of Foundation Model

Foundation models matter: federated learning for multi-center tuberculosis diagnosis via adaptive regularization and model-contrastive learning

Foundation models matter: federated learning for multi-center tuberculosis diagnosis via adaptive regularization and model-contrastive learning

A novel method for predicting coarse aggregate particle size distribution based on segment anything model and machine learning

Particle size distribution (PSD) is an important index property of granular materials. The conventional mechanical sieve analysis for PSD determination is labor-intensive, time-consuming, and inefficient. With the development of computer vision, image-based gradation prediction methods gradually emerge. However, previous studies mainly focused on images where individual aggregate particles are separated from each other without overlapping (i.e., not densely packed together), thus limiting their applicability. This study proposed a novel framework to predict the PSD of densely packed coarse aggregate particles, which is based on visual foundation model and machine learning with appropriate feature engineering. First, the segment anything model, which is one of the visual foundation models, was used for the instance segmentation of the densely packed aggregate particles. Then, the feature engineering was implemented with the segmentation result. The major quantitative indicators related to particle sizes were extracted from each mask, such as the area, the diameter of minimum enclosing circle, and the major axes of the equivalent ellipse of particle contours. Subsequently, the cumulative distribution curves of these indicators were plotted, followed by the data sampling operation on these curves to generate the feature vectors. The artificial neural network (ANN) model was then trained for gradation prediction with each of the feature vectors and the data points extracted from PSD curves used as the inputs and data labels, respectively. The model-training dataset consisted of 8160 different sets of digitally generated particle samples with known PSDs. Finally, verification test results of additional five different groups of digital particle samples and real coarse aggregate samples with known PSDs show that the root mean squared error (RMSE) of predicted aggregate sizes is 1.79 mm and 1.86 mm for digital and real aggregate particle samples, respectively. Therefore, the efficacy and feasibility of the proposed method was validated in terms of predicting the PSD of densely packed coarse aggregates from two-dimensional image information. As compared to the Mask RCNN combined with empirical formula of which the RMSE values were 5.12 mm and 5.84 mm for digital and real aggregate particle samples, respectively, the proposed framework demonstrates improved particle segmentation applicability and better PSD prediction accuracy.