Virtual Staining Research Articles

Restoring H&E stain in faded slides via phase-to-color virtual staining in near-infrared

Histological stains, such as hematoxylin and eosin, tend to fade over time, compromising subsequent analysis accuracy. Traditional methods of restoring stain color in faded samples involve physical re-staining, which is time-consuming and expensive and may damage tissue samples. In addition, digital post-processing techniques, such as color normalization, face limitations when dealing with highly faded slides. To address this, we propose the non-invasive phase-to-color “virtual re-staining” framework. This approach utilizes a trained generative adversarial network with label-free quantitative phase imaging, capturing the intrinsic physiochemical properties of histological samples. It employs multi-channel Fourier ptychographic microscopy to generate pixel-wise paired phase and color images in a high-throughput manner. To streamline data generation, near-infrared illumination is used to mitigate the impact of absorption variations in faded and stained samples, eliminating the need for repetitive data acquisition and potential physical alterations in samples. Our trained network yields comparable or better results to other digitally staining methods, successfully demonstrating the re-staining of approximately decade-old faded slides archived in hospital storage.

PST-Diff: Achieving High-Consistency Stain Transfer by Diffusion Models With Pathological and Structural Constraints.

Histopathological examinations heavily rely on hematoxylin and eosin (HE) and immunohistochemistry (IHC) staining. IHC staining can offer more accurate diagnostic details but it brings significant financial and time costs. Furthermore, either re-staining HE-stained slides or using adjacent slides for IHC may compromise the accuracy of pathological diagnosis due to information loss. To address these challenges, we develop PST-Diff, a method for generating virtual IHC images from HE images based on diffusion models, which allows pathologists to simultaneously view multiple staining results from the same tissue slide. To maintain the pathological consistency of the stain transfer, we propose the asymmetric attention mechanism (AAM) and latent transfer (LT) module in PST-Diff. Specifically, the AAM can retain more local pathological information of the source domain images, while ensuring the model's flexibility in generating virtual stained images that highly confirm to the target domain. Subsequently, the LT module transfers the implicit representations across different domains, effectively alleviating the bias introduced by direct connection and further enhancing the pathological consistency of PST-Diff. Furthermore, to maintain the structural consistency of the stain transfer, the conditional frequency guidance (CFG) module is proposed to precisely control image generation and preserve structural details according to the frequency recovery process. To conclude, the pathological and structural consistency constraints provide PST-Diff with effectiveness and superior generalization in generating stable and functionally pathological IHC images with the best evaluation score. In general, PST-Diff offers prospective application in clinical virtual staining and pathological image analysis.

Stain transformation using Mueller matrix guided generative adversarial networks.

Recently, virtual staining techniques have attracted more and more attention, which can help bypass the chemical staining process of traditional histopathological examination, saving time and resources. Meanwhile, as an emerging tool to characterize specific tissue structures in a label-free manner, the Mueller matrix microscopy can supplement more structural information that may not be apparent in bright-field images. In this Letter, we propose the Mueller matrix guided generative adversarial networks (MMG-GAN). By integrating polarization information provided by the Mueller matrix microscopy, the MMG-GAN enables the effective transformation of input H&E-stained images into corresponding Masson trichrome (MT)-stained images. The experimental results demonstrate the accuracy of the generated images by MMG-GAN and reveal the potential for more stain transformation tasks by incorporating the Mueller matrix polarization information, laying the foundation for future polarimetry-assisted digital pathology.

Deep learning-based virtual staining, segmentation, and classification in label-free photoacoustic histology of human specimens

In pathological diagnostics, histological images highlight the oncological features of excised specimens, but they require laborious and costly staining procedures. Despite recent innovations in label-free microscopy that simplify complex staining procedures, technical limitations and inadequate histological visualization are still problems in clinical settings. Here, we demonstrate an interconnected deep learning (DL)-based framework for performing automated virtual staining, segmentation, and classification in label-free photoacoustic histology (PAH) of human specimens. The framework comprises three components: (1) an explainable contrastive unpaired translation (E-CUT) method for virtual H&E (VHE) staining, (2) an U-net architecture for feature segmentation, and (3) a DL-based stepwise feature fusion method (StepFF) for classification. The framework demonstrates promising performance at each step of its application to human liver cancers. In virtual staining, the E-CUT preserves the morphological aspects of the cell nucleus and cytoplasm, making VHE images highly similar to real H&E ones. In segmentation, various features (e.g., the cell area, number of cells, and the distance between cell nuclei) have been successfully segmented in VHE images. Finally, by using deep feature vectors from PAH, VHE, and segmented images, StepFF has achieved a 98.00% classification accuracy, compared to the 94.80% accuracy of conventional PAH classification. In particular, StepFF’s classification reached a sensitivity of 100% based on the evaluation of three pathologists, demonstrating its applicability in real clinical settings. This series of DL methods for label-free PAH has great potential as a practical clinical strategy for digital pathology.

Feature Fusion GAN Based Virtual Staining on Plant Microscopy Images.

Virtual staining of microscopy specimens using GAN-based methods could resolve critical concerns of manual staining process as displayed in recent studies on histopathology images. However, most of these works use basic-GAN framework ignoring microscopy image characteristics and their performance were evaluated based on structural and error statistics (SSIM and PSNR) between synthetic and ground-truth without considering any color space although virtual staining deals with color transformation. Besides, major aspects of staining, like color, contrast, focus, image-realness etc. were totally ignored. However, modifications of GAN architecture for virtual staining might be suitable by incorporating microscopy image features. Further, its successful implementation need to be examined by considering various aspects of staining process. Therefore, we designed, a new feature-fusion-GAN for virtual staining followed by performance assessment by framing a state-of-the-art multi-evaluation framework that includes numerous metrics in -qualitative (based on histogram-correlation of color and brightness); quantitative (SSIM and PSNR); focus aptitude (Brenner metrics and Spectral-Moments); and influence on perception (semantic perceptual influence score). For, experimental validation cell boundaries were highlighted by two different staining reagents, Safranin-O and Toluidine-Blue-O on plant microscopy images of potato tuber. We evaluated virtually stained image quality w.r.t ground-truth in RGB and YCbCr color spaces based on defined metrics and results are found very consistent. Further, impact of feature fusion has been demonstrated. Collectively, this study could be a baseline towards guiding architectural upgrading of deep pipelines for virtual staining of diverse microscopy modalities followed by future benchmark methodology or protocols.

Evaluation of tumor budding with virtual panCK stains generated by novel multi-model CNN framework

As the global incidence of cancer continues to rise rapidly, the need for swift and precise diagnoses has become increasingly pressing. Pathologists commonly rely on H&E-panCK stain pairs for various aspects of cancer diagnosis, including the detection of occult tumor cells and the evaluation of tumor budding. Nevertheless, conventional chemical staining methods suffer from notable drawbacks, such as time-intensive processes and irreversible staining outcomes. The virtual stain technique, leveraging generative adversarial network (GAN), has emerged as a promising alternative to chemical stains. This approach aims to transform biopsy scans (often H&E) into other stain types. Despite achieving notable progress in recent years, current state-of-the-art virtual staining models confront challenges that hinder their efficacy, particularly in achieving accurate staining outcomes under specific conditions. These limitations have impeded the practical integration of virtual staining into diagnostic practices. To address the goal of producing virtual panCK stains capable of replacing chemical panCK, we propose an innovative multi-model framework. Our approach involves employing a combination of Mask-RCNN (for cell segmentation) and GAN models to extract cytokeratin distribution from chemical H&E images. Additionally, we introduce a tailored dynamic GAN model to convert H&E images into virtual panCK stains, integrating the derived cytokeratin distribution. Our framework is motivated by the fact that the unique pattern of the panCK is derived from cytokeratin distribution. As a proof of concept, we employ our virtual panCK stains to evaluate tumor budding in 45 H&E whole-slide images taken from breast cancer-invaded lymph nodes . Through thorough validation by both pathologists and the QuPath software, our virtual panCK stains demonstrate a remarkable level of accuracy. In stark contrast, the accuracy of state-of-the-art single cycleGAN virtual panCK stains is negligible. To our best knowledge, this is the first instance of a multi-model virtual panCK framework and the utilization of virtual panCK for tumor budding assessment. Our framework excels in generating dependable virtual panCK stains with significantly improved efficiency, thereby considerably reducing turnaround times in diagnosis. Furthermore, its outcomes are easily comprehensible even to pathologists who may not be well-versed in computer technology. We firmly believe that our framework has the potential to advance the field of virtual stain, thereby making significant strides towards improved cancer diagnosis.

Clinical-Grade Validation of an Autofluorescence Virtual Staining System With Human Experts and a Deep Learning System for Prostate Cancer

The tissue diagnosis of adenocarcinoma and intraductal carcinoma of the prostate includes Gleason grading of tumor morphology on the hematoxylin and eosin stain and immunohistochemistry markers on the prostatic intraepithelial neoplasia-4 stain (CK5/6, P63, and AMACR). In this work, we create an automated system for producing both virtual hematoxylin and eosin and prostatic intraepithelial neoplasia-4 immunohistochemistry stains from unstained prostate tissue using a high-throughput hyperspectral fluorescence microscope and artificial intelligence and machine learning. We demonstrate that the virtual stainer models can produce high-quality images suitable for diagnosis by genitourinary pathologists. Specifically, we validate our system through extensive human review and computational analysis, using a previously validated Gleason scoring model, and an expert panel, on a large data set of test slides. This study extends our previous work on virtual staining from autofluorescence, demonstrates the clinical utility of this technology for prostate cancer, and exemplifies a rigorous standard of qualitative and quantitative evaluation for digital pathology.

A Dual-Modal, Label-Free Raman Imaging Method for Rapid Virtual Staining of Large-Area Breast Cancer Tissue Sections.

As one of the most common cancers, accurate, rapid, and simple histopathological diagnosis is very important for breast cancer. Raman imaging is a powerful technique for label-free analysis of tissue composition and histopathology, but it suffers from slow speed when applied to large-area tissue sections. In this study, we propose a dual-modal Raman imaging method that combines Raman mapping data with microscopy bright-field images to achieve virtual staining of breast cancer tissue sections. We validate our method on various breast tissue sections with different morphologies and biomarker expressions and compare it with the golden standard of histopathological methods. The results demonstrate that our method can effectively distinguish various types and components of tissues, and provide staining images comparable to stained tissue sections. Moreover, our method can improve imaging speed by up to 65 times compared to general spontaneous Raman imaging methods. It is simple, fast, and suitable for clinical applications.

In vivo organoid growth monitoring by stimulated Raman histology

Patient-derived tumor organoids have emerged as a crucial tool for assessing the efficacy of chemotherapy and conducting preclinical drug screenings. However, the conventional histological investigation of these organoids necessitates their devitalization through fixation and slicing, limiting their utility to a single-time analysis. Here, we use stimulated Raman histology (SRH) to demonstrate non-destructive, label-free virtual staining of 3D organoids, while preserving their viability and growth. This novel approach provides contrast similar to conventional staining methods, allowing for the continuous monitoring of organoids over time. Our results demonstrate that SRH transforms organoids from one-time use products into repeatable models, facilitating the efficient selection of effective drug combinations. This advancement holds promise for personalized cancer treatment, allowing for the dynamic assessment and optimization of chemotherapy treatments in patient-specific contexts.

Deep learning-based virtual H& E staining from label-free autofluorescence lifetime images

Label-free autofluorescence lifetime is a unique feature of the inherent fluorescence signals emitted by natural fluorophores in biological samples. Fluorescence lifetime imaging microscopy (FLIM) can capture these signals enabling comprehensive analyses of biological samples. Despite the fundamental importance and wide application of FLIM in biomedical and clinical sciences, existing methods for analysing FLIM images often struggle to provide rapid and precise interpretations without reliable references, such as histology images, which are usually unavailable alongside FLIM images. To address this issue, we propose a deep learning (DL)-based approach for generating virtual Hematoxylin and Eosin (H&E) staining. By combining an advanced DL model with a contemporary image quality metric, we can generate clinical-grade virtual H&E-stained images from label-free FLIM images acquired on unstained tissue samples. Our experiments also show that the inclusion of lifetime information, an extra dimension beyond intensity, results in more accurate reconstructions of virtual staining when compared to using intensity-only images. This advancement allows for the instant and accurate interpretation of FLIM images at the cellular level without the complexities associated with co-registering FLIM and histology images. Consequently, we are able to identify distinct lifetime signatures of seven different cell types commonly found in the tumour microenvironment, opening up new opportunities towards biomarker-free tissue histology using FLIM across multiple cancer types.

Mantis: high-throughput 4D imaging and analysis of the molecular and physical architecture of cells.

High-throughput dynamic imaging of cells and organelles is essential for understanding complex cellular responses. We report Mantis, a high-throughput 4D microscope that integrates two complementary, gentle, live-cell imaging technologies: remote-refocus label-free microscopy and oblique light-sheet fluorescence microscopy. Additionally, we report shrimPy, an open-source software for high-throughput imaging, deconvolution, and single-cell phenotyping of 4D data. Using Mantis and shrimPy, we achieved high-content correlative imaging of molecular dynamics and the physical architecture of 20 cell lines every 15 minutes over 7.5 hours. This platform also facilitated detailed measurements of the impacts of viral infection on the architecture of host cells and host proteins. The Mantis platform can enable high-throughput profiling of intracellular dynamics, long-term imaging and analysis of cellular responses to perturbations, and live-cell optical screens to dissect gene regulatory networks.

Noninvasive virtual biopsy using micro-registered optical coherence tomography (OCT) in human subjects.

Histological hematoxylin and eosin-stained (H&E) tissue sections are used as the gold standard for pathologic detection of cancer, tumor margin detection, and disease diagnosis. Producing H&E sections, however, is invasive and time-consuming. While deep learning has shown promise in virtual staining of unstained tissue slides, true virtual biopsy requires staining of images taken from intact tissue. In this work, we developed a micron-accuracy coregistration method [micro-registered optical coherence tomography (OCT)] that can take a two-dimensional (2D) H&E slide and find the exact corresponding section in a 3D OCT image taken from the original fresh tissue. We trained a conditional generative adversarial network using the paired dataset and showed high-fidelity conversion of noninvasive OCT images to virtually stained H&E slices in both 2D and 3D. Applying these trained neural networks to in vivo OCT images should enable physicians to readily incorporate OCT imaging into their clinical practice, reducing the number of unnecessary biopsy procedures.

The virtual staining method by quantitative phase imaging for label free lymphocytes based on self-supervised iteration cycle-consistent adversarial networks.

Quantitative phase imaging (QPI) provides 3D structural and morphological information for label free living cells. Unfortunately, this quantitative phase information cannot meet doctors' diagnostic requirements of the clinical "gold standard," which displays stained cells' pathological states based on 2D color features. To make QPI results satisfy the clinical "gold standard," the virtual staining method by QPI for label free lymphocytes based on self-supervised iteration Cycle-Consistent Adversarial Networks (CycleGANs) is proposed herein. The 3D phase information of QPI is, therefore, trained and transferred to a kind of 2D "virtual staining" image that is well in agreement with "gold standard" results. To solve the problem that unstained QPI and stained "gold standard" results cannot be obtained for the same label free living cell, the self-supervised iteration for the CycleGAN deep learning algorithm is designed to obtain a trained stained result as the ground truth for error evaluation. The structural similarity index of our virtual staining experimental results for 8756 lymphocytes is 0.86. Lymphocytes' area errors after converting to 2D virtual stained results from 3D phase information are less than 3.59%. The mean error of the nuclear to cytoplasmic ratio is 2.69%, and the color deviation from the "gold standard" is less than 6.67%.

Efficient Supervised Pretraining of Swin-Transformer for Virtual Staining of Microscopy Images.

Fluorescence staining is an important technique in life science for labeling cellular constituents. However, it also suffers from being time-consuming, having difficulty in simultaneous labeling, etc. Thus, virtual staining, which does not rely on chemical labeling, has been introduced. Recently, deep learning models such as transformers have been applied to virtual staining tasks. However, their performance relies on large-scale pretraining, hindering their development in the field. To reduce the reliance on large amounts of computation and data, we construct a Swin-transformer model and propose an efficient supervised pretraining method based on the masked autoencoder (MAE). Specifically, we adopt downsampling and grid sampling to mask 75% of pixels and reduce the number of tokens. The pretraining time of our method is only 1/16 compared with the original MAE. We also design a supervised proxy task to predict stained images with multiple styles instead of masked pixels. Additionally, most virtual staining approaches are based on private datasets and evaluated by different metrics, making a fair comparison difficult. Therefore, we develop a standard benchmark based on three public datasets and build a baseline for the convenience of future researchers. We conduct extensive experiments on three benchmark datasets, and the experimental results show the proposed method achieves the best performance both quantitatively and qualitatively. In addition, ablation studies are conducted, and experimental results illustrate the effectiveness of the proposed pretraining method. The benchmark and code are available at https://github.com/birkhoffkiki/CAS-Transformer.

Abstract 6185: Generative AI for virtual HE-staining of whole slide images of unstained breast cancer tissue

Abstract Introduction Hematoxylin and eosin (HE) is the standard stain used in histology to make tissue visible to the human eye by highlighting certain cellular and tissue structures, and it is a technique that is widely used in the diagnosis of cancer and other pathologies. Chemical staining, however, is irreversible, making the tissue unusable for subsequent measurements, such as spatial transcriptomics. Here we utilize generative AI method based on pix2pix image-to-image translation to generate virtual HE-staining for whole slide images (WSIs) acquired of unstained tissue with brightfield microscopy and perform a thorough histological evaluation for the feasibility in breast cancer diagnostics. Materials and Methods We optimized sample preparation and imaging setup for virtual staining purposes, and developed a custom generative adversarial network architecture for learning the virtual staining from paires samples of unstained and H&E stained tissue. Here, we focus on the utility of virtual staining in breast cancer diagnostics, using a set of breast cancer samples for acquiring whole slide images from unstained tissue before H&E staining and from reference H&E stained tissue after chemical staining. Hold-out set of sample pairs are left for validation, allowing us to evaluate the virtual staining performance for a vast array of tissue components and to examine the potential shortcomings in staining reproduction. We use a comprehensive set of quantitative metrics both on pixel and object level to evaluate virtual staining quality. In addition, we perform thorough visual evaluation of histological feasibility by histology experts to examine the computational staining. Results We demonstrate that by careful optimization of both sample preparation and imaging workflow, as well as the computational methods, generative adversarial networks can be used for virtual staining of whole slide images of breast cancer tissue acquired from unstained tissue using regular bright field microscopy. We analyzed the virtual staining performance quantitatively and visually, and highlight the potential of the method through successful cases and demonstrate the applicability in breast cancer diagnostics, as well as discuss the challenges and shortcomings of virtual staining for clinical samples. Conclusions We demonstrate the potential of virtual HE staining for clinical histopathology of breast cancer tissue. Notably, our virtual staining based on generative AI shows promise towards more sustainable and streamlined sample processing and staining process in digital pathology. Citation Format: Leena Latonen, Umair Khan, Sonja Koivukoski, Johan Hartman, Pekka Ruusuvuori. Generative AI for virtual HE-staining of whole slide images of unstained breast cancer tissue [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 6185.

Abstract 914: Combination analysis of tumor-associated collagen frameworks and tumor immune phenotype of lung carcinomas using virtual staining

Abstract An emerging predictive parameter of immunotherapy response is the patient’s tumor immune status, generally classified as “inflamed”, "immune excluded” and “desert”. Historically, classification was performed semi-quantitatively with somewhat subjective parameters as evidenced by the recent Delphi Workshop consensus. There are a variety of approaches to classifying immune status, most based upon analysis of H&E images (counting tumor infiltrating lymphocytes, TILs), or counting subpopulations of immune cells using immunologic staining. However, use of multiplexed or multiple stains has recently been explored to better quantify the counts and types of immune cells in individual tissue compartments. While the immune microenvironment represents an important parameter in predicting patient response, other regions such as the extracellular matrix (ECM) may yield complementary information, as collagen-rich ECMs may present a barrier to drug diffusion and the orientation of fibers may direct the migration of malignant cells. Here we present a proof of concept virtual staining enhanced image analysis pipeline, which converts autofluorescence signals from a single tissue section into virtual H&E and Masson’s Trichrome along with virtual detection of pan cytokeratin (AE1/AE3/PCK26) and CD45 (LCA) positive cells. We apply image analysis to the panel of virtual stains to study the spatial and case-by-case heterogeneity of tumor collagen frameworks and immune phenotype. Using a standard slide scanner (Axio Scan Z1, Zeiss), multiple autofluorescence images were captured from unstained sections (4-6um thick) of lung tissue. The virtual staining was performed by four deep neural networks trained in a supervised learning fashion. Select chemical stains were performed on previously scanned tissues and reviewed by pathologists side-by-side with virtual stains to ensure consistency and quality control. Four perfectly registered WSI virtual stain images were generated from each tissue section. The multi-stain results were rendered from a variety of lung cancers, including tissue microarray slides. Image analysis was performed using HALO (Indica Labs) and custom python scripts. We identified unique areas of tumor and adjacent stroma with heterogeneous immune phenotype and collagen characteristics, suggesting that an interplay might exist that could be utilized to improve patient selection or prognosis when deploying advanced AI tools. Future work includes applying this virtual stain technique retrospectively to cases with known immunotherapy treatment responses to determine the prognostic significance of the combined immune/ECM phenotype. Citation Format: Serge Alexanian, Yair Rivenson, Ning Xuan, Brian Cone, Zihang Fang, Sean Meyering, Luis A. Carvalho, David Palacios, Raymond Kozikowski. Combination analysis of tumor-associated collagen frameworks and tumor immune phenotype of lung carcinomas using virtual staining [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 914.

Abstract 7401: High-level multiplexed virtual immunofluorescence staining using holotomography and deep learning

Abstract Multiplexed immunofluorescence (IF) staining plays a crucial role in cancer research, allowing simultaneous visualization of multiple biomarkers to elucidate complex cellular interactions within the tumor microenvironment. However, the conventional multiplexed IF approach has inherent limitations, characterized by labor-intensiveness and a constrained capacity for high-level multiplexing. These challenges hinder its scalability and comprehensive analysis of intricate molecular landscapes.To overcome the limitations of conventional multiplexed IF, we propose a novel approach that integrates holotomography (HT) and deep learning. HT is a label-free imaging technique that quantitatively measures the refractive index distribution of the specimen without any staining. To provide subcellular specificity to the label-free HT images, HT is integrated with virtual staining. Virtual staining is a deep learning-based framework that converts label-free images to the desired stained images, in this case, from HT images to multiplexed IF images. This synergistic combination of HT and deep learning may address the current limitations of multiplexed IF protocols.Our proposed methodology aims at high-level multiplexed virtual IF staining from label-free cancer tissue slides using HT and deep learning. This approach not only eliminates the need for chemical staining but also enhances the level of multiplexing through computational staining. Specifically, we acquire HT and fluorescence images of stained tissue slides for training. Utilizing this paired dataset, we then employ supervised learning to create a neural network that can map between label-free HT images and corresponding fluorescence images. Subsequently, the trained neural network is applied to label-free tissue slides, inferring fluorescence images without the need for chemical staining.We believe the proposed virtual staining framework will contribute to a more efficient and scalable paradigm for cancer research, facilitating a deeper understanding of tumor biology, and advancing the field of precision medicine. Citation Format: Juyeon Park, Su-Jin Shin, Geon Kim, Kwang Suk Lee, Ji Eun Heo, YongKeun Park. High-level multiplexed virtual immunofluorescence staining using holotomography and deep learning [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 7401.

Abstract 3517: A deep learning-based virtual staining system with multimodal-attention for precision medicine

Abstract Purpose: This study introduces a novel multimodal-attention-based virtual mIF staining (MAS) system designed for efficient, reliable virtual mIF staining from label-free fluorescence images. Our goal was to overcome the performance bottleneck and time-cost limitations associated with traditional mIF techniques and thereby enhance their clinical utility. Method and Materials: Our approach involved developing a cutting-edge MAS model. This model is built upon a sophisticated end-to-end generative convolutional neural network (CNN) architecture. It ingeniously leverages autofluorescence and 4',6-diamidino-2-phenylindole (DAPI) slides as inputs to generate mIF images. To achieve this, we equipped the model with feature extractors enhanced by pretrained masked auto-encoders (MAEs) and a self-attention combination strategy. These components worked harmoniously to extract antigen-label-related features and precisely locate specific cells within the images. Results: In our comprehensive study, we engaged 94 gastric cancer patients, utilizing the MAS system for automated virtual mIF staining of seven biomarkers, namely CD3, CD20, FOXP3, PD1, CD8, CD163, and PDL1, in both cancerous and non-cancerous tissues. Importantly, the MAS-produced virtual mIF stains matched the quality of traditional manual stains. Furthermore, we validated the prognostic accuracy for gastric cancer using these virtual mIF images, demonstrating their ability to provide clinical information that is as reliable and valuable as that obtained from manually stained mIF images. Conclusions: Our study identifies the MAS system as an important advancement in mIF staining, enhancing personalized medicine with its efficiency and quality. This tool holds significant potential to enable personalized medicine efficiently and cost-effectively, streamlining the process of diagnosing diseases, making prognoses, and developing treatment plans. Quantitative Comparison of Ablation Experiments for MAS system in predicting multiple biomarkers Biomarkers Index U-Net (Avg±Std) ReU-Net (Avg±Std) Att-ReU-Net (Avg±Std) MAS (Avg±Std) CD3 PSNR 27.053±4.10 27.419±4.15 28.073±4.00 28.546±4.40 CD3 SSIM 0.733±0.07 0.724±0.07 0.728±0.05 0.742±0.07 CD20 PSNR 28.835±4.66 29.244±4.76 28.672±4.52 29.243±4.99 CD20 SSIM 0.921±0.058 0.921±0.057 0.915±0.051 0.923±0.051 FOXP3 PSNR 31.769±3.29 31.838±3.10 31.098±3.20 31.862±3.11 FOXP3 SSIM 0.613±0.11 0.618±0.11 0.617±0.112 0.617±0.12 PD1 PSNR 27.908±4.47 27.950±4.43 30.283±4.37 31.717±4.17 PD1 SSIM 0.654±0.13 0.659±0.13 0.694±0.12 0.726±0.11 Citation Format: Zixia Zhou, Yuming Jiang, Ruijiang Li, Lei Xing. A deep learning-based virtual staining system with multimodal-attention for precision medicine [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 3517.

Virtual histological staining of unlabeled autopsy tissue

Traditional histochemical staining of post-mortem samples often confronts inferior staining quality due to autolysis caused by delayed fixation of cadaver tissue, and such chemical staining procedures covering large tissue areas demand substantial labor, cost and time. Here, we demonstrate virtual staining of autopsy tissue using a trained neural network to rapidly transform autofluorescence images of label-free autopsy tissue sections into brightfield equivalent images, matching hematoxylin and eosin (H&E) stained versions of the same samples. The trained model can effectively accentuate nuclear, cytoplasmic and extracellular features in new autopsy tissue samples that experienced severe autolysis, such as COVID-19 samples never seen before, where the traditional histochemical staining fails to provide consistent staining quality. This virtual autopsy staining technique provides a rapid and resource-efficient solution to generate artifact-free H&E stains despite severe autolysis and cell death, also reducing labor, cost and infrastructure requirements associated with the standard histochemical staining.

A Virtual Staining Method Based on Self-Supervised GAN for Fourier Ptychographic Microscopy Colorful Imaging

Fourier ptychographic microscopy (FPM) is a computational imaging technology that has endless vitality and application potential in digital pathology. Colored pathological image analysis is the foundation of clinical diagnosis, basic research, and most biomedical problems. However, the current colorful FPM reconstruction methods are time-inefficient, resulting in poor image quality due to optical interference and reconstruction errors. This paper combines coloring and FPM to propose a self-supervised generative adversarial network (GAN) for FPM color reconstruction. We design a generator based on the efficient channel residual (ECR) block to adaptively obtain efficient cross-channel interaction information in a lightweight manner, and we introduce content-consistency loss to learn the high-frequency information of the image and improve the image quality of the staining. Furthermore, the effectiveness of our proposed method is demonstrated through objective indicators and visual evaluations.