Challenging Datasets Research Articles

Zero-Shot Aerial Object Detection with Visual Description Regularization

Existing object detection models are mainly trained on large-scale labeled datasets. However, annotating data for novel aerial object classes is expensive since it is time-consuming and may require expert knowledge. Thus, it is desirable to study label-efficient object detection methods on aerial images. In this work, we propose a zero-shot method for aerial object detection named visual Description Regularization, or DescReg. Concretely, we identify the weak semantic-visual correlation of the aerial objects and aim to address the challenge with prior descriptions of their visual appearance. Instead of directly encoding the descriptions into class embedding space which suffers from the representation gap problem, we propose to infuse the prior inter-class visual similarity conveyed in the descriptions into the embedding learning. The infusion process is accomplished with a newly designed similarity-aware triplet loss which incorporates structured regularization on the representation space. We conduct extensive experiments with three challenging aerial object detection datasets, including DIOR, xView, and DOTA. The results demonstrate that DescReg significantly outperforms the state-of-the-art ZSD methods with complex projection designs and generative frameworks, e.g., DescReg outperforms best reported ZSD method on DIOR by 4.5 mAP on unseen classes and 8.1 in HM. We further show the generalizability of DescReg by integrating it into generative ZSD methods as well as varying the detection architecture. Codes will be released at https://github.com/zq-zang/DescReg.

CamoDiffusion: Camouflaged Object Detection via Conditional Diffusion Models

Camouflaged Object Detection (COD) is a challenging task in computer vision due to the high similarity between camouflaged objects and their surroundings. Existing COD methods struggle with nuanced object boundaries and overconfident incorrect predictions. In response, we propose a new paradigm that treats COD as a conditional mask-generation task leveraging diffusion models. Our method, dubbed CamoDiffusion, employs the denoising process to progressively refine predictions while incorporating image conditions. Due to the stochastic sampling process of diffusion, our model is capable of sampling multiple possible predictions, avoiding the problem of overconfident point estimation. Moreover, we develop specialized network architecture, training, and sampling strategies, to enhance the model’s expressive power, refinement capabilities and suppress overconfident mis-segmentations, thus aptly tailoring the diffusion model to the demands of COD. Extensive experiments on three COD datasets attest to the superior performance of our model compared to existing state-of-the-art methods, particularly on the most challenging COD10K dataset, where our approach achieves 0.019 in terms of MAE. Codes and models are available at https://github.com/Rapisurazurite/CamoDiffusion.

Keypoint Fusion for RGB-D Based 3D Hand Pose Estimation

Previous 3D hand pose estimation methods primarily rely on a single modality, either RGB or depth, and the comprehensive utilization of the dual modalities has not been extensively explored. RGB and depth data provide complementary information and thus can be fused to enhance the robustness of 3D hand pose estimation. However, there exist two problems for applying existing fusion methods in 3D hand pose estimation: redundancy of dense feature fusion and ambiguity of visual features. First, pixel-wise feature interactions introduce high computational costs and ineffective calculations of invalid pixels. Second, visual features suffer from ambiguity due to color and texture similarities, as well as depth holes and noise caused by frequent hand movements, which interferes with modeling cross-modal correlations. In this paper, we propose Keypoint-Fusion for RGB-D based 3D hand pose estimation, which leverages the unique advantages of dual modalities to mutually eliminate the feature ambiguity, and performs cross-modal feature fusion in a more efficient way. Specifically, we focus cross-modal fusion on sparse yet informative spatial regions (i.e. keypoints). Meanwhile, by explicitly extracting relatively more reliable information as disambiguation evidence, depth modality provides 3D geometric information for RGB feature pixels, and RGB modality complements the precise edge information lost due to the depth noise. Keypoint-Fusion achieves state-of-the-art performance on two challenging hand datasets, significantly decreasing the error compared with previous single-modal methods.

Image as a Language: Revisiting Scene Text Recognition via Balanced, Unified and Synchronized Vision-Language Reasoning Network

Scene text recognition is inherently a vision-language task. However, previous works have predominantly focused either on extracting more robust visual features or designing better language modeling. How to effectively and jointly model vision and language to mitigate heavy reliance on a single modality remains a problem. In this paper, aiming to enhance vision-language reasoning in scene text recognition, we present a balanced, unified and synchronized vision-language reasoning network (BUSNet). Firstly, revisiting the image as a language by balanced concatenation along length dimension alleviates the issue of over-reliance on vision or language. Secondly, BUSNet learns an ensemble of unified external and internal vision-language model with shared weight by masked modality modeling (MMM). Thirdly, a novel vision-language reasoning module (VLRM) with synchronized vision-language decoding capacity is proposed. Additionally, BUSNet achieves improved performance through iterative reasoning, which utilizes the vision-language prediction as a new language input. Extensive experiments indicate that BUSNet achieves state-of-the-art performance on several mainstream benchmark datasets and more challenge datasets for both synthetic and real training data compared to recent outstanding methods. Code and dataset will be available at https://github.com/jjwei66/BUSNet.

HyperEditor: Achieving Both Authenticity and Cross-Domain Capability in Image Editing via Hypernetworks

Editing real images authentically while also achieving cross-domain editing remains a challenge. Recent studies have focused on converting real images into latent codes and accomplishing image editing by manipulating these codes. However, merely manipulating the latent codes would constrain the edited images to the generator's image domain, hindering the attainment of diverse editing goals. In response, we propose an innovative image editing method called HyperEditor, which utilizes weight factors generated by hypernetworks to reassign the weights of the pre-trained StyleGAN2's generator. Guided by CLIP's cross-modal image-text semantic alignment, this innovative approach enables us to simultaneously accomplish authentic attribute editing and cross-domain style transfer, a capability not realized in previous methods. Additionally, we ascertain that modifying only the weights of specific layers in the generator can yield an equivalent editing result. Therefore, we introduce an adaptive layer selector, enabling our hypernetworks to autonomously identify the layers requiring output weight factors, which can further improve our hypernetworks' efficiency. Extensive experiments on abundant challenging datasets demonstrate the effectiveness of our method.

Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language

Given an untrimmed video and a sentence query, video moment retrieval using language (VMR) aims to locate a target query-relevant moment. Since the untrimmed video is overlong, almost all existing VMR methods first sparsely down-sample each untrimmed video into multiple fixed-length video clips and then conduct multi-modal interactions with the query feature and expensive clip features for reasoning, which is infeasible for long real-world videos that span hours. Since the video is downsampled into fixed-length clips, some query-related frames may be filtered out, which will blur the specific boundary of the target moment, take the adjacent irrelevant frames as new boundaries, easily leading to cross-modal misalignment and introducing both boundary-bias and reasoning-bias. To this end, in this paper, we propose an efficient approach, SpotVMR, to trim the query-relevant clip. Besides, our proposed SpotVMR can serve as plug-and-play module, which achieves efficiency for state-of-the-art VMR methods while maintaining good retrieval performance. Especially, we first design a novel clip search model that learns to identify promising video regions to search conditioned on the language query. Then, we introduce a set of low-cost semantic indexing features to capture the context of objects and interactions that suggest where to search the query-relevant moment. Also, the distillation loss is utilized to address the optimization issues arising from end-to-end joint training of the clip selector and VMR model. Extensive experiments on three challenging datasets demonstrate its effectiveness.

Abstract 3511: iQC: machine-learning-driven prediction of surgery reveals systematic confounds in cancer whole slide images from hospitals by protocol

Abstract Problem: This 21st century has produced an eruption of research using AI for the detection and diagnosis of cancer. Yet, an often-unspoken core premise in this field of computational pathology is that a glass slide suitably represents the patient’s disease. Here, we report systematic confounds may dominate slides from a medical center, such that slides are unsuitable for diagnosis. Methods: We mathematically define high quality data as a whole slide image set where the patient’s surgery may be accurately predicted by an automated system. Our system “iQC” accurately distinguished biopsies (i.e. thin strands of tissue) from nonbiopsies, e.g. transurethral resections (TURPs) or prostatectomies, only when the data appeared high quality, e.g. bright histopathology stains and few artifacts. Thus, when the data are of high quality, iQC (i) accurately classifies pixels as tissue, (ii) accurately generates stats that describe the distribution of tissue, and (iii) accurately predicts surgical procedure from those stats. We compare iQC against the published HistoQC tool. Results: iQC holds all data to the same objective quality standard. We validate this standard in five Veterans Affairs Medical Centers (VAMCs) and the public Automated Gleason Grading Challenge (AGGC) dataset. For the surgery prediction task, we report an AUROC of 0.9966-1.000 at VAMCs that produced high quality data and AUROC=0.9824 for AGGC. In contrast, we report AUROC=0.7115 at the VAMC that produced poor quality data. A pathologist found poor quality may be explained by faded histopathology stains and VAMC protocol differences. Supporting this, iQC's novel stain strength statistic finds this VAMC had weaker stains (p < 2.2e-16, two-tailed Wilcoxon rank-sum test; Cohen's d=1.208) than the VAMC that contributed most of the slides. Additionally, iQC recommended only 2 of 3736 (0.005%) VAMC slides for review due to inadequate tissue. In contrast, HistoQC in its default configuration excluded 89.9% of VAMC slides because tissue was not detected, but we reduced this to 16.7% with our custom HistoQC configuration. Conclusion: Our surgery prediction AUROC may be a quantitative indicator positively associated with data quality for a dataset. Unless data are poor quality, iQC accurately locates tissue in slides and excludes few slides. iQC is, to our knowledge, the first automated system in computational pathology that validates quality against objective evidence, e.g. surgical procedure data available in the EHR/LIMS, which requires no efforts or annotations from anatomic pathologists. Citation Format: Andrew J. Schaumberg, Michael S. Lewis, Ramin Nazarian, Ananta Wadhwa, Nathanael Kane, Graham Turner, Purushotham Karnam, Poornima Devineni, Nicholas Wolfe, Randall Kintner, Matthew B. Rettig, Beatrice S. Knudsen, Isla P. Garraway, Saiju Pyarajan. iQC: machine-learning-driven prediction of surgery reveals systematic confounds in cancer whole slide images from hospitals by protocol [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 3511.

Efficient networks for textureless feature registration via free receptive field

In the real-world scenarios, challenges such as changes in viewpoint, variations in lighting conditions, and the presence of blur effects are pervasive issues faced in the field of image registration, making the rapid and accurate establishment of correspondences between images exceedingly difficult. Presently, encoder-based methods often entail larger parameter sizes and longer execution times. Additionally, when dealing with low-texture images, reliable keypoint registration becomes particularly challenging, leading to further increases in registration costs. To address such issues, this study proposes a novel Efficient Feature Registration Network (EFRNet) that integrates low-frequency texture information with high-frequency contour information to achieve efficient and stable feature registration. The proposed EFRNet possesses several significant advantages. Firstly, the introduced Free Receptive Field (FreeRF) module dynamically adapts to allocate extractors for each feature map and discards irrelevant feature information, thereby facilitating the extraction of features at different frequencies and achieving model lightweighting. Secondly, the Add Edge Margin Global Feature Registrant (AemGFR), incorporating a cross-attention architecture, flexibly adjusts feature interaction processes to meet the requirements of accurate registration. Moreover, by introducing a Gaussian distribution of edge values, the AemGFR effectively accelerates model convergence and overall efficiency. Throughout the entire algorithm, the FreeRF module is fully utilized to further enhance feature extraction capability while reducing the parameter count of the network during the testing phase, achieving lightweight feature registration. Extensive experimentation has demonstrated that our EFRNet exhibits significant advantages over state-of-the-art feature registration methods on three challenging datasets. Importantly, our method achieves faster inference speeds across different sensor devices.

Semantic segmentation of progressive micro-cracking in polymer composites using Attention U-Net architecture

The present study delivers a methodology for investigating the gradual damage development in a carbon fibre-reinforced cross-ply polymer composite during a sequence of thermo-mechanical loadings with the help of X-ray computed tomography. The procedure allows an in-depth analysis of the occurrence and nature of the multiple cracks that form within layers oriented perpendicular, or transverse, to the loading direction. This is achieved by using Attention U-Net architecture for semantic segmentation of the transverse cracks. The model shows promising results, through an ability to identify all the transverse cracks and reflect the damage progression. The described method provides a robust routine for analysing challenging polymer composite tomographic datasets.

Predicting Knee Joint Contact Forces During Normal Walking Using Kinematic Inputs With a Long-Short Term Neural Network.

Knee joint contact forces are commonly estimated via surrogate measures (i.e., external knee adduction moments or musculoskeletal modeling). Despite its capabilities, modeling is not optimal for clinicians or persons with limited experience. The purpose of this study was to design a novel prediction method for knee joint contact forces that is simplistic in terms of required inputs. This study included marker trajectories and instrumented knee forces during normal walking from the "Grand Challenge" (n = 6) and "CAMS" (n = 2) datasets. Inverse kinematics were used to derive stance phase hip (sagittal, frontal, transverse), knee (sagittal, frontal), ankle (sagittal), and trunk (frontal) kinematics. A long-short term memory network (LSTM) was created using matlab to predict medial and lateral knee force waveforms using combinations of the kinematics. The Grand Challenge and CAMS datasets trained and tested the network, respectively. Musculoskeletal modeling forces were derived using static optimization and joint reaction tools in OpenSim. Waveform accuracy was determined as the proportion of variance and root-mean-square error between network predictions and in vivo data. The LSTM network was highly accurate for medial forces (R2 = 0.77, RMSE = 0.27 BW) and required only frontal hip and knee and sagittal hip and ankle kinematics. Modeled medial force predictions were excellent (R2 = 0.77, RMSE = 0.33 BW). Lateral force predictions were poor for both methods (LSTM R2 = 0.18, RMSE = 0.08 BW; modeling R2 = 0.21, RMSE = 0.54 BW). The designed LSTM network outperformed most reports of musculoskeletal modeling, including those reached in this study, revealing knee joint forces can accurately be predicted by using only kinematic input variables.

MS2Rescore 3.0 Is a Modular, Flexible, and User-Friendly Platform to Boost Peptide Identifications, as Showcased with MS Amanda 3.0.

Rescoring of peptide-spectrum matches (PSMs) has emerged as a standard procedure for the analysis of tandem mass spectrometry data. This emphasizes the need for software maintenance and continuous improvement for such algorithms. We introduce MS2Rescore 3.0, a versatile, modular, and user-friendly platform designed to increase peptide identifications. Researchers can install MS2Rescore across various platforms with minimal effort and benefit from a graphical user interface, a modular Python API, and extensive documentation. To showcase this new version, we connected MS2Rescore 3.0 with MS Amanda 3.0, a new release of the well-established search engine, addressing previous limitations on automatic rescoring. Among new features, MS Amanda now contains additional output columns that can be used for rescoring. The full potential of rescoring is best revealed when applied on challenging data sets. We therefore evaluated the performance of these two tools on publicly available single-cell data sets, where the number of PSMs was substantially increased, thereby demonstrating that MS2Rescore offers a powerful solution to boost peptide identifications. MS2Rescore's modular design and user-friendly interface make data-driven rescoring easily accessible, even for inexperienced users. We therefore expect the MS2Rescore to be a valuable tool for the wider proteomics community. MS2Rescore is available at https://github.com/compomics/ms2rescore.

PFD-Net: Pyramid Fourier Deformable Network for medical image segmentation

Medical image segmentation is crucial for accurately locating lesion regions and assisting doctors in diagnosis. However, most existing methods fail to effectively utilize both local details and global semantic information in medical image segmentation, resulting in the inability to effectively capture fine-grained content such as small targets and irregular boundaries. To address this issue, we propose a novel Pyramid Fourier Deformable Network (PFD-Net) for medical image segmentation, which leverages the strengths of CNN and Transformer. The PFD-Net first utilizes PVTv2-based Transformer as the primary encoder to capture global information and further enhances both local and global feature representations with the Fast Fourier Convolution Residual (FFCR) module. Moreover, PFD-Net further proposes the Dilated Deformable Refinement (DDR) module to enhance the model’s capacity to comprehend global semantic structures of shape-diverse targets and their irregular boundaries. Lastly, Cross-Level Fusion Block with deformable convolution (CLFB) is proposed to combine the decoded feature maps from the final Residual Decoder Block (DDR) with local features from the CNN auxiliary encoder branch, improving the network’s ability to perceive targets resembling the surrounding structures. Extensive experiments were conducted on nine publicly medical image datasets for five types of segmentation tasks including polyp, abdominal, cardiac, gland cells and nuclei. The qualitative and quantitative results demonstrate that PFD-Net outperforms existing state-of-the-art methods in various evaluation metrics, and achieves the highest performance of mDice with the value of 0.826 on the most challenging dataset (ETIS), which is 1.8% improvement compared to the previous best-performing HSNet and 3.6% improvement compared to the next-best PVT-CASCADE. Codes are available at https://github.com/ChaorongYang/PFD-Net.

CellViT: Vision Transformers for precise cell segmentation and classification

Nuclei detection and segmentation in hematoxylin and eosin-stained (H&E) tissue images are important clinical tasks and crucial for a wide range of applications. However, it is a challenging task due to nuclei variances in staining and size, overlapping boundaries, and nuclei clustering. While convolutional neural networks have been extensively used for this task, we explore the potential of Transformer-based networks in combination with large scale pre-training in this domain. Therefore, we introduce a new method for automated instance segmentation of cell nuclei in digitized tissue samples using a deep learning architecture based on Vision Transformer called CellViT. CellViT is trained and evaluated on the PanNuke dataset, which is one of the most challenging nuclei instance segmentation datasets, consisting of nearly 200,000 annotated nuclei into 5 clinically important classes in 19 tissue types. We demonstrate the superiority of large-scale in-domain and out-of-domain pre-trained Vision Transformers by leveraging the recently published Segment Anything Model and a ViT-encoder pre-trained on 104 million histological image patches — achieving state-of-the-art nuclei detection and instance segmentation performance on the PanNuke dataset with a mean panoptic quality of 0.50 and an F1-detection score of 0.83. The code is publicly available at https://github.com/TIO-IKIM/CellViT.

Progressive deep snake for instance boundary extraction in medical images

Boundary extraction is meaningful in medical image analysis since it explicitly extracts the tissue/lesions boundary coordinates, which benefits the follow-up diagnosis processes. However, accurately extracting human tissue/lesions is challenging due to unclear target boundaries or pseudo boundaries, which could lead to inferior performance of existing boundary extraction and/or semantic segmentation methods. We propose a Progressive Deep Snake (PDS), which is designed with a multi-step contour evolution (MCE) and a sequential label generation module (SLGM) to explicitly extract the smooth and continuous boundary of different tissue/lesions even if they are unclear, with irregular shape, or occluded by pseudo boundaries. (1) MCE is designed to initialize and evolve the contours to the target boundaries, where a multi-step deep supervision strategy is designed to improve flexibility and accuracy. (2) SLGM is designed to generate easy-to-difficult supervision signals for the contour evolution, which provides targets of moderate difficulty in each step for better guiding the contour evolution. Experiments on a challenging verbal segment datasets show that our method achieves state-of-the-art performance (MR_AVBCE dataset: mIoU: 0.9234, mDice: 0.9598, mBoundF: 0.9241, mWcov: 0.9704). Our code is available at https://github.com/sysu19351118/PDS-pytorch.

Optimization of Using Multiple Machine Learning Approaches in Atrial Fibrillation Detection Based on a Large-Scale Data Set of 12-Lead Electrocardiograms: Cross-Sectional Study.

Atrial fibrillation (AF) represents a hazardous cardiac arrhythmia that significantly elevates the risk of stroke and heart failure. Despite its severity, its diagnosis largely relies on the proficiency of health care professionals. At present, the real-time identification of paroxysmal AF is hindered by the lack of automated techniques. Consequently, a highly effective machine learning algorithm specifically designed for AF detection could offer substantial clinical benefits. We hypothesized that machine learning algorithms have the potential to identify and extract features of AF with a high degree of accuracy, given the intricate and distinctive patterns present in electrocardiogram (ECG) recordings of AF. This study aims to develop a clinically valuable machine learning algorithm that can accurately detect AF and compare different leads' performances of AF detection. We used 12-lead ECG recordings sourced from the 2020 PhysioNet Challenge data sets. The Welch method was used to extract power spectral features of the 12-lead ECGs within a frequency range of 0.083 to 24.92 Hz. Subsequently, various machine learning techniques were evaluated and optimized to classify sinus rhythm (SR) and AF based on these power spectral features. Furthermore, we compared the effects of different frequency subbands and different lead selections on machine learning performances. The light gradient boosting machine (LightGBM) was found to be the most effective in classifying AF and SR, achieving an average F1-score of 0.988 across all ECG leads. Among the frequency subbands, the 0.083 to 4.92 Hz range yielded the highest F1-score of 0.985. In interlead comparisons, aVR had the highest performance (F1=0.993), with minimal differences observed between leads. In conclusion, this study successfully used machine learning methodologies, particularly the LightGBM model, to differentiate SR and AF based on power spectral features derived from 12-lead ECGs. The performance marked by an average F1-score of 0.988 and minimal interlead variation underscores the potential of machine learning algorithms to bolster real-time AF detection. This advancement could significantly improve patient care in intensive care units as well as facilitate remote monitoring through wearable devices, ultimately enhancing clinical outcomes.

Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery

PurposeDepth estimation in robotic surgery is vital in 3D reconstruction, surgical navigation and augmented reality visualization. Although the foundation model exhibits outstanding performance in many vision tasks, including depth estimation (e.g., DINOv2), recent works observed its limitations in medical and surgical domain-specific applications. This work presents a low-ranked adaptation (LoRA) of the foundation model for surgical depth estimation.MethodsWe design a foundation model-based depth estimation method, referred to as Surgical-DINO, a low-rank adaptation of the DINOv2 for depth estimation in endoscopic surgery. We build LoRA layers and integrate them into DINO to adapt with surgery-specific domain knowledge instead of conventional fine-tuning. During training, we freeze the DINO image encoder, which shows excellent visual representation capacity, and only optimize the LoRA layers and depth decoder to integrate features from the surgical scene.ResultsOur model is extensively validated on a MICCAI challenge dataset of SCARED, which is collected from da Vinci Xi endoscope surgery. We empirically show that Surgical-DINO significantly outperforms all the state-of-the-art models in endoscopic depth estimation tasks. The analysis with ablation studies has shown evidence of the remarkable effect of our LoRA layers and adaptation.ConclusionSurgical-DINO shed some light on the successful adaptation of the foundation models into the surgical domain for depth estimation. There is clear evidence in the results that zero-shot prediction on pre-trained weights in computer vision datasets or naive fine-tuning is not sufficient to use the foundation model in the surgical domain directly.

Multi-residual 2D network integrating spatial correlation for whole heart segmentation

Whole heart segmentation (WHS) has significant clinical value for cardiac anatomy, modeling, and analysis of cardiac function. This study aims to address the WHS accuracy on cardiac CT images, as well as the fast inference speed and low graphics processing unit (GPU) memory consumption required by practical clinical applications. Thus, we propose a multi-residual two-dimensional (2D) network integrating spatial correlation for WHS. The network performs slice-by-slice segmentation on three-dimensional cardiac CT images in a 2D encoder–decoder manner. In the network, a convolutional long short-term memory skip connection module is designed to perform spatial correlation feature extraction on the feature maps at different resolutions extracted by the sub-modules of the pre-trained ResNet-based encoder. Moreover, a decoder based on the multi-residual module is designed to analyze the extracted features from the perspectives of multi-scale and channel attention, thereby accurately delineating the various substructures of the heart. The proposed method is verified on a dataset of the multi-modality WHS challenge, an in-house WHS dataset, and a dataset of the abdominal organ segmentation challenge. The dice, Jaccard, average symmetric surface distance, Hausdorff distance, inference time, and maximum GPU memory of the WHS are 0.914, 0.843, 1.066 mm, 15.778 mm, 9.535 s, and 1905 MB, respectively. The proposed network has high accuracy, fast inference speed, minimal GPU memory consumption, strong robustness, and good generalization. It can be deployed to clinical practical applications for WHS and can be effectively extended and applied to other multi-organ segmentation fields. The source code is publicly available at https://github.com/nancy1984yan/MultiResNet-SC.

Brain MR image simulation for deep learning based medical image analysis networks

Background and ObjectiveAs large sets of annotated MRI data are needed for training and validating deep learning based medical image analysis algorithms, the lack of sufficient annotated data is a critical problem. A possible solution is the generation of artificial data by means of physics-based simulations. Existing brain simulation data is limited in terms of anatomical models, tissue classes, fixed tissue characteristics, MR sequences and overall realism. MethodsWe propose a realistic simulation framework by incorporating patient-specific phantoms and Bloch equations-based analytical solutions for fast and accurate MRI simulations. A large number of labels are derived from open-source high-resolution T1w MRI data using a fully automated brain classification tool. The brain labels are taken as ground truth (GT) on which MR images are simulated using our framework. Moreover, we demonstrate that the T1w MR images generated from our framework along with GT annotations can be utilized directly to train a 3D brain segmentation network. To evaluate our model further on larger set of real multi-source MRI data without GT, we compared our model to existing brain segmentation tools, FSL-FAST and SynthSeg. ResultsOur framework generates 3D brain MRI for variable anatomy, sequence, contrast, SNR and resolution. The brain segmentation network for WM/GM/CSF trained only on T1w simulated data shows promising results on real MRI data from MRBrainS18 challenge dataset with a Dice scores of 0.818/0.832/0.828. On OASIS data, our model exhibits a close performance to FSL, both qualitatively and quantitatively with a Dice scores of 0.901/0.939/0.937. ConclusionsOur proposed simulation framework is the initial step towards achieving truly physics-based MRI image generation, providing flexibility to generate large sets of variable MRI data for desired anatomy, sequence, contrast, SNR, and resolution. Furthermore, the generated images can effectively train 3D brain segmentation networks, mitigating the reliance on real 3D annotated data.

Generation of PAS-stained images of glomerular tissue units using a generative adversarial network with spectral normalization colorization method

In recent years, the pathological diagnosis of glomerular diseases typically involves the study of glomerular his-to pathology by specialized pathologists, who analyze tissue sections stained with Periodic Acid-Schiff (PAS) to assess tissue and cellular abnormalities. In recent years, the rapid development of generative adversarial networks composed of generators and discriminators has led to further developments in image colorization tasks. In this paper, we present a generative adversarial network by Spectral Normalization colorization designed for color restoration of grayscale images depicting glomerular cell tissue elements. The network consists of two structures: the generator and the discriminator. The generator incorporates a U-shaped decoder and encoder network to extract feature information from input images, extract features from Lab color space images, and predict color distribution. The discriminator network is responsible for optimizing the generated colorized images by comparing them with real stained images. On the Human Biomolecular Atlas Program (HubMAP)—Hacking the Kidney FTU segmentation challenge dataset, we achieved a peak signal-to-noise ratio of 29.802 dB, along with high structural similarity results as other colorization methods. This colorization method offers an approach to add color to grayscale images of glomerular cell tissue units. It facilitates the observation of physiological information in pathological images by doctors and patients, enabling better pathological-assisted diagnosis of certain kidney diseases.

Domain Generalization with Small Data

In this work, we propose to tackle the problem of domain generalization in the context of insufficient samples. Instead of extracting latent feature embeddings based on deterministic models, we propose to learn a domain-invariant representation based on the probabilistic framework by mapping each data point into probabilistic embeddings. Specifically, we first extend empirical maximum mean discrepancy (MMD) to a novel probabilistic MMD that can measure the discrepancy between mixture distributions (i.e., source domains) consisting of a series of latent distributions rather than latent points. Moreover, instead of imposing the contrastive semantic alignment (CSA) loss based on pairs of latent points, a novel probabilistic CSA loss encourages positive probabilistic embedding pairs to be closer while pulling other negative ones apart. Benefiting from the learned representation captured by probabilistic models, our proposed method can marriage the measurement on the distribution over distributions (i.e., the global perspective alignment) and the distribution-based contrastive semantic alignment (i.e., the local perspective alignment). Extensive experimental results on three challenging medical datasets show the effectiveness of our proposed method in the context of insufficient data compared with state-of-the-art methods.