Image-based Modeling Research Articles

FireVoxel: Interactive Software for Multi-Modality Analysis of Dynamic Medical Images.

This article provides an overview of the FireVoxel software for quantitative analysis of medical images and its applications in the field. We describe FireVoxel's user interface, multi-layer design, dynamic parametric models, and several turn-key workflows. Additionally, we discuss its application in recent imaging projects. We outline basic image analysis tools such as segmentation, non-uniformity correction, and coregistration through a pictorial overview, with a focus on deformable coregistration and motion correction. Several example workflows and image-based dynamic modeling are also highlighted. Furthermore, we analyze peer-reviewed studies that utilized FireVoxel for image processing, categorizing published papers based on body structures/organs, image processing methods, and imaging modalities. For comparison, we searched the Ovid MEDLINE database to assess the general use of medical image analysis software. FireVoxel is used by over 3000 users worldwide, with 528 articles, including 413 in English, published in the past 15years. MRI is the most commonly used imaging modality (78.2%), followed by CT (14.5%) and PET (7.3%). The most frequently used methods are dynamic modeling, segmentation, texture analysis, and coregistration. FireVoxel is commonly used in abdominal and genitourinary imaging studies, where it appears to fill a niche due to the lack of alternative software. The search of the Ovid MEDLINE suggests that quantitative medical imaging studies, on the other hand, focus on the brain and cardiovascular system. FireVoxel offers an effective set of quantitative tools, particularly for abdominal and genitourinary imaging, likely due to its ability to manage patient motion and correct for MR artifacts. The software is especially valuable for processing dynamic studies. The steady increase in publications utilizing FireVoxel reflects growing interest in this software and its relevance for image-based research.

Reliable and streamlined model setup for digital twin assessment of fracture healing.

In large animal models of bone fracture repair, postmortem torsional testing is commonly used to assess healing biomechanics. Bending and axial tests are physiologically relevant, but much less commonly performed. Virtual torsional testing using image-based finite element models has been validated to postmortem bench tests, but its predictive value for capturing whole-bone mechanics and fracture healing quality under other physiologically relevant loading modes has not yet been established. Accordingly, the purpose of this study was to evaluate the association between mechanical biomarkers derived from virtual torsion, axial, and bending tests under strict alignment and malalignment conditions. Computed tomography (CT) scans from 24 intact and operated sheep tibiae and 29 human tibial fractures were used to create digital twins that were subjected to torsion, axial, and bending tests. The results indicated that torsional rigidity is a strong surrogate for bending flexural rigidity in both ovine and human bones. Torsional rigidity and axial stiffness were strongly correlated in the ovine data, but only moderately in human fractures due to the complex fracture patterns. Axial testing was highly prone to stiffness estimation errors as high as 50% if the applied load and anatomic axis were not perfectly aligned. In contrast, torsional rigidity had errors <1.3% for all malalignment scenarios. Based on this study, virtual torsional rigidity is the recommended summary mechanical biomarker of bone healing because it captures variations in healing biomechanics that are present in other loading modes with a simple setup that is insensitive to alignment error.

Association of echocardiographic radiomics-based features with cardiotoxicity effect in breast cancer patients from the CARDIOCARE project

Abstract Introduction Improvements in breast cancer (BC) treatment including chemotherapy have increased significantly the BC survivors. However, higher BC survival rates have been compromised by life-threatening collateral toxic effects. Effects such as cardiomyopathy and heart failure, described as cardiotoxicity, have become main causes of comorbidity and mortality, having a great impact on patients’ quality of life. Purpose Leverage radiomics-based analysis to explore innovative, non-invasive echocardiography biomarkers that can associate with cardiotoxicity. Method A cohort of 28 prospective patients (above 60 years old, 10 cardiotoxic) of the EU-funded CARDIOCARE project [1] undergoing cardiotoxic antineoplasmatic therapy was utilized to investigate potential association between imaging features calculated from the end diastolic (ED) and end systolic (ES) apical 4- and 2-chamber views and patients’ cardiotoxicity status as defined by the project's cardiologists using imaging data (Left Ventricle Ejection Fraction , Global Longitudinal Strain) and blood test markers (high sensitive Troponin I and Nt-pro-Bnp). Imaging data were collected at baseline visit (start of the treatment) and 3 months after. A pre-trained deep neural network [2] was initially utilized to automatically segment the myocardium (MY) and the left ventricle (LV) from the ED and ES 4- and 2-chamber static views both at baseline (T0) and 3-month follow-up (T1). More than 1500 imaging features were calculated from all images of the MY and LV using radiomics, a state-of-the-art technique to elicit hidden image patterns using histogram, shape and textural features. Apart from radiomics at T0 and T1, time-dependent features (delta-radiomics) were also provided as the relative change in radiomics values between T0 and T1, aiming to reveal potential changes at the MY and LV areas through treatment. Cardiotoxicity status was defined at T1. Results A univariate analysis using the Wilcoxon rank-sum test identified a subset of 40 radiomics features (20 delta-radiomics and 20 from T1) to be considered statistically significant (p-value&amp;lt;0.05) in differentiating cardiotoxic from non-cardiotoxic patients. Histogram and shape-based characteristics showed no association with cardiotoxicity status. Additionally, radiomics-based features at T0 were not statistically significant. All significant features corresponded to textural properties of the MY, indicating that the heterogeneity of the MY can be a significant factor for cardiotoxicity. 17 out of 20 T1 features were calculated from the 4-chamber view of the MY (19/20 from the ED). 15 delta-radiomics textural features were extracted from the 2-chamber view of the MY region at the ES. Conclusion Heterogeneity of the myocardium quantified by radiomics has potential diagnostic impact in assessing cardiotoxicity response, paving the way for the development of image-based risk stratification models for the early detection of cardiac toxicity.

Predicting On-Road Air Pollution Coupling Street View Images and Machine Learning: A Quantitative Analysis of the Optimal Strategy.

Integrating mobile monitoring data with street view images (SVIs) holds promise for predicting local air pollution. However, algorithms, sampling strategies, and image quality introduce extra errors due to a lack of reliable references that quantify their effects. To bridge this gap, we employed 314 taxis to monitor NO, NO2, PM2.5, and PM10, and extracted features from ∼382,000 SVIs at multiple angles (0°, 90°, 180°, 270°) and buffer radii (100-500 m). Additionally, three typical machine learning algorithms were compared with SVI-based land-used regression (LUR) model to explore their performances. Generally, machine learning methods outperform linear LUR, with the ranking: random forest > XGBoost > neural network > LUR. Averaging strategy is an effective method to avoid bias of insufficient feature capture. Therefore, the optimal sampling strategy is to integrating multiple viewing angles at a 100-m buffer, which achieved absolute errors mostly less than 2.5 μg/m3 or ppb. Besides, overexposure, blur, and underexposure led to image misjudgments and incorrect identifications, causing an overestimation of road features and underestimation of human-activity features. These findings enhance understanding and offer valuable support for developing image-based air quality models and other SVI-related research.

Dataset for developing deep learning models to assess crack width and self-healing progress in concrete

The presented dataset comes from an experimental study on the autogenous self-healing of high-strength concrete and the development of deep learning metasensor for crack width assessment and self-healing evaluation. Concrete specimens were prepared, matured, cracked, and exposed to self-healing. High-resolution scanning of the specimen surface and scale-invariant image processing were performed, multiple grid lines crossing cracks were established, and brightness degree profiles along grid lines were extracted. Then, reference measurements of the crack widths were obtained by an operator. The dataset comprises 19,098 records of brightness profiles, reference measurements, and benchmark measurements by deep learning and analytic models. The source images, stacked and marked with grid lines, are provided. The considerable number of brightness profiles coupled with manual reference measurements make the dataset well suited for developing an image-based deep CNN models or analytic algorithms for assessing crack widths in concrete. The technical validation study explored three factors that affect crack measurement: the specimen position in relation to the scanner, the surface moisture level, and the operator performing manual measurements.

Automatic summarization of cooking videos using transfer learning and transformer-based models

The proliferation of cooking videos on the internet these days necessitates the conversion of these lengthy video contents into concise text recipes. Many online platforms now have a large number of cooking videos, in which, there is a challenge for viewers to extract comprehensive recipes from lengthy visual content. Effective summary is necessary in order to translate the abundance of culinary knowledge found in videos into text recipes that are easy to read and follow. This will make the cooking process easier for individuals who are searching for precise step by step cooking instructions. Such a system satisfies the needs of a broad spectrum of learners while also improving accessibility and user simplicity. As there is a growing need for easy-to-follow recipes made from cooking videos, researchers are looking on the process of automated summarization using advanced techniques. One such approach is presented in our work, which combines simple image-based models, audio processing, and GPT-based models to create a system that makes it easier to turn long culinary videos into in-depth recipe texts. A systematic workflow is adopted in order to achieve the objective. Initially, Focus is given for frame summary generation which employs a combination of two convolutional neural networks and a GPT-based model. A pre-trained CNN model called Inception-V3 is fine-tuned with food image dataset for dish recognition and another custom-made CNN is built with ingredient images for ingredient recognition. Then a GPT based model is used to combine the results produced by the two CNN models which will give us the frame summary in the desired format. Subsequently, Audio summary generation is tackled by performing Speech-to-text functionality in python. A GPT-based model is then used to generate a summary of the resulting textual representation of audio in our desired format. Finally, to refine the summaries obtained from visual and auditory content, Another GPT-based model is used which combines the output of the frame summary and audio summary modules and give the final enhanced summary. By minimizing the complications involved with traditional and sophisticated methodologies, this research helps with the development of a straightforward but efficient cooking video summarization system. The results achieved in the work are on par with the existing work in the respective field which demonstrates comparable performance and efficacy in converting cooking videos into detailed recipe texts.

Combining a Risk Factor Score Designed From Electronic Health Records With a Digital Cytology Image Scoring System to Improve Bladder Cancer Detection: Proof-of-Concept Study.

To reduce the mortality related to bladder cancer, efforts need to be concentrated on early detection of the disease for more effective therapeutic intervention. Strong risk factors (eg, smoking status, age, professional exposure) have been identified, and some diagnostic tools (eg, by way of cystoscopy) have been proposed. However, to date, no fully satisfactory (noninvasive, inexpensive, high-performance) solution for widespread deployment has been proposed. Some new models based on cytology image classification were recently developed and bring good perspectives, but there are still avenues to explore to improve their performance. Our team aimed to evaluate the benefit of combining the reuse of massive clinical data to build a risk factor model and a digital cytology image-based model (VisioCyt) for bladder cancer detection. The first step relied on designing a predictive model based on clinical data (ie, risk factors identified in the literature) extracted from the clinical data warehouse of the Rennes Hospital and machine learning algorithms (logistic regression, random forest, and support vector machine). It provides a score corresponding to the risk of developing bladder cancer based on the patient's clinical profile. Second, we investigated 3 strategies (logistic regression, decision tree, and a custom strategy based on score interpretation) to combine the model's score with the score from an image-based model to produce a robust bladder cancer scoring system. We collected 2 data sets. The first set, including clinical data for 5422 patients extracted from the clinical data warehouse, was used to design the risk factor-based model. The second set was used to measure the models' performances and was composed of data for 620 patients from a clinical trial for which cytology images and clinicobiological features were collected. With this second data set, the combination of both models obtained areas under the curve of 0.82 on the training set and 0.83 on the test set, demonstrating the value of combining risk factor-based and image-based models. This combination offers a higher associated risk of cancer than VisioCyt alone for all classes, especially for low-grade bladder cancer. These results demonstrate the value of combining clinical and biological information, especially to improve detection of low-grade bladder cancer. Some improvements will need to be made to the automatic extraction of clinical features to make the risk factor-based model more robust. However, as of now, the results support the assumption that this type of approach will be of benefit to patients.

PET-Assessed Metabolic Tumor Volume Across the Spectrum of Solid-Organ Malignancies: A Review of the Literature.

Solid-organ malignancies represent a significant disease burden and remain one of the leading causes of death globally. In the past few decades, the rapid evolution of imaging modalities has shifted the paradigm towards image-based precision medicine, especially in the care of patients with solid-organ malignancies. Metabolic tumor volume (MTV) is one such semi-quantitative parameter obtained from positron emission tomography (PET) imaging with 18F-fluorodeoxyglucose (FDG) that has been shown to have significant implications in the clinical oncology setting. Across various solid tumor malignancies, including lung cancer, head and neck cancer, breast cancer, esophageal cancer, and colorectal cancer, the current literature has demonstrated an association between MTV and various clinical outcomes. MTV may be used in conjunction with several existing and established clinical parameters to help inform risk stratification and treatment strategies and predict outcomes in cancer. Optimizing such volumetric parameters is paramount for advancing efforts to advance cancer care for our patients. While such advancements are made, it is important to investigate and address the limitations of MTV, including variability in terms of measurement methods, a lack of standardized cut-off values, and the impact of inherent tumor heterogeneity. Despite these limitations, which can precipitate challenges in standardization, MTV as a prognostic factor has great potential and opens an avenue for the future integration of technology into an image-based precision medicine model of care for cancer patients. This article serves as a narrative review and explores the utility and limitations of PET-MTV in various settings of solid-organ malignancy.

Optimal integration of forest inventory data and aerial image-based canopy height models for forest stand management

Optimal integration of forest inventory data and aerial image-based canopy height models for forest stand management

Modeling Fibrous Tissue in Vascular Fluid-Structure Interaction: A Morphology-Based Pipeline and Biomechanical Significance.

Modeling fibrous tissue for vascular fluid-structure interaction analysis poses significant challenges due to the lack of effective tools for preparing simulation data from medical images. This limitation hinders the physiologically realistic modeling of vasculature and its use in clinical settings. Leveraging an established lumen modeling strategy, we propose a comprehensive pipeline for generating thick-walled artery models. A specialized mesh generation procedure is developed to ensure mesh continuity across the lumen and wall interface. Exploiting the centerline information, a series of procedures are introduced for generating local basis vectors within the arterial wall. The procedures are tailored to handle thick-walled tissues where basis vectors may exhibit transmural variations. Additionally, we propose methods for accurately identifying the centerline in multi-branched vessels and bifurcating regions. These modeling approaches are algorithmically implementable, rendering them readily integrable into mainstream cardiovascular modeling software. The developed fiber generation method is evaluated against the strategy using linear elastostatics analysis, demonstrating that the proposed approach yields satisfactory fiber definitions in the considered benchmark. Finally, we examine the impact of anisotropic arterial wall models on the vascular fluid-structure interaction analysis through numerical examples, employing the neo-Hookean model for comparative purposes. The first case involves an idealized curved geometry, while the second studies an image-based abdominal aorta model. Our numerical results reveal that the deformation and stress distribution are critically related to the constitutive model of the wall, whereas hemodynamic factors are less sensitive to the wall model. This work paves the way for more accurate image-based vascular modeling and enhances the prediction of arterial behavior under physiologically realistic conditions.

The Classification of Metastatic Spine Cancer and Spinal Compression Fractures by Using CNN and SVM Techniques.

Metastatic spine cancer can cause pain and neurological issues, making it challenging to distinguish from spinal compression fractures using magnetic resonance imaging (MRI). To improve diagnostic accuracy, this study developed artificial intelligence (AI) models to differentiate between metastatic spine cancer and spinal compression fractures in MRI images. MRI data from Gyeongsang National University Hospital, collected from January 2019 to April 2022, were processed using Otsu's binarization and Canny edge detection algorithms. Using these preprocessed datasets, convolutional neural network (CNN) and support vector machine (SVM) models were built. The T1-weighted image-based CNN model demonstrated high sensitivity (1.00) and accuracy (0.98) in identifying metastatic spine cancer, particularly with data processed by Otsu's binarization and Canny edge detection, achieving exceptional performance in detecting cancerous cases. This approach highlights the potential of preprocessed MRI data for AI-assisted diagnosis, supporting clinical applications in distinguishing metastatic spine cancer from spinal compression fractures.

Intravascular Ultrasound Image-Based Finite Element Modeling Approach for Quantifying In Vivo Mechanical Properties of Human Coronary Artery.

Quantifying the mechanical properties of coronary arterial walls could provide meaningful information for the diagnosis, management, and treatment of coronary artery diseases. Since patient-specific coronary samples are not available for patients requiring continuous monitoring, direct experimental testing of vessel material properties becomes impossible. Current coronary models typically use material parameters from available literature, leading to significant mechanical stress/strain calculation errors. Here, we would introduce a finite element model-based updating approach (FEMBUA) to quantify patient-specific in vivo material properties of coronary arteries based on medical images. In vivo cine intravascular ultrasound (IVUS) and virtual histology (VH)-IVUS images of coronary arteries were acquired from a patient with coronary artery disease. Cine IVUS images showing the vascular movement over one cardiac cycle were segmented, and two IVUS frames with maximum and minimum lumen circumferences were selected to represent the coronary geometry under systolic and diastolic pressure conditions, respectively. VH-IVUS image was also segmented to obtain the vessel contours, and a layer thickness of 0.05 cm was added to the VH-IVUS contours to reconstruct the coronary geometry. A computational finite element model was created with an anisotropic Mooney-Rivlin material model used to describe the vessel's mechanical properties and pulsatile blood pressure conditions prescribed to the coronary luminal surface to make it contract and expand. Then, an iterative updating approach was employed to determine the material parameters of the anisotropic Mooney-Rivlin model by matching minimum and maximum lumen circumferences from the computational finite element model with those from cine IVUS images. This image-based finite element model-based updating approach could be successfully extended to determine the material properties of arterial walls in various vascular beds and holds the potential for risk assessment of cardiovascular diseases.

Fast-Vid2Vid++: Spatial-Temporal Distillation for Real-Time Video-to-Video Synthesis.

Video-to-Video synthesis (Vid2Vid) gains remarkable performance in generating a photo-realistic video from a sequence of semantic maps, such as segmentation, sketch and pose. However, this pipeline is heavily limited to high computational cost and long inference latency, mainly attributed to two essential factors: 1) network architecture parameters, 2) sequential data stream. Recently, the parameters of image-based generative models have been significantly reduced via more efficient network architectures. Existing methods mainly focus on slimming network architectures but ignore the size of the sequential data stream. Moreover, due to the lack of temporal coherence, image-based compression is not sufficient for the compression of the video task. In this paper, we present a spatial-temporal hybrid distillation compression framework, Fast-Vid2Vid++, which focuses on knowledge distillation of the teacher network and the data stream of generative models on both space and time. Fast-Vid2Vid++ makes the first attempt at time dimension to transfer hierarchical features and time coherence knowledge to reduce computational resources and accelerate inference. Specifically, we compress the data stream spatially and reduce the temporal redundancy. We distill the knowledge of the hierarchical features and the final response from the teacher network to the student network in high-resolution and full-time domains. We transfer the long-term dependencies of the features and video frames to the student model. After the proposed spatial-temporal hybrid knowledge distillation (Spatial-Temporal-HKD), our model can synthesize high-resolution key-frames using the low-resolution data stream. Finally, Fast-Vid2Vid++ interpolates intermediate frames by motion compensation with slight latency and generates full-length sequences with motion-aware inference (MAI). On standard benchmarks, Fast-Vid2Vid++ achieves a real-time performance of 30-59 FPS and saves 28-35× computational cost on a single V100 GPU.

Experimental validation of an image-based dynamic pore-network model for spontaneous imbibition in sandstones

Experimental validation of an image-based dynamic pore-network model for spontaneous imbibition in sandstones

Development and Validation of Dynamic 5-Year Breast Cancer Risk Model Using Repeated Mammograms.

Current image-based long-term risk prediction models do not fully use previous screening mammogram images. Dynamic prediction models have not been investigated for use in routine care. We analyzed a prospective WashU clinic-based cohort of 10,099 cancer-free women at entry (between November 3, 2008 and February 2012). Follow-up through 2020 identified 478 pathology-confirmed breast cancers (BCs). The cohort included 27% Black women. An external validation cohort (Emory) included 18,360 women screened from 2013, followed through 2020. This included 42% Black women and 332 pathology-confirmed BC excluding those diagnosed within 6 months of screening. We trained a dynamic model using repeated screening mammograms at WashU to predict 5-year risk. This opportunistic screening service presented a range of mammogram images for each woman. We applied the model to the external validation data to evaluate discrimination performance (AUC) and calibrated to US SEER. Using 3 years of previous mammogram images available at the current screening visit, we obtained a 5-year AUC of 0.80 (95% CI, 0.78 to 0.83) in the external validation. This represents a significant improvement over the current visit mammogram AUC 0.74 (95% CI, 0.71 to 0.77; P < .01) in the same women. When calibrated, a risk ratio of 21.1 was observed comparing high (>4%) to very low (<0.3%) 5-year risk. The dynamic model classified 16% of the cohort as high risk among whom 61% of all BCs were diagnosed. The dynamic model performed comparably in Black and White women. Adding previous screening mammogram images improves 5-year BC risk prediction beyond static models. It can identify women at high risk who might benefit from supplemental screening or risk-reduction strategies.

Efficacy of a whole slide image-based prediction model for lymph node metastasis in T1 colorectal cancer: A systematic review.

Accurate stratification of the risk of lymph node metastasis (LNM) following endoscopic resection of submucosal invasive (T1) colorectal cancer (CRC) is imperative for determining the necessity for additional surgery. In this systematic review, we evaluated the efficacy of prediction of LNM by artificial intelligence (AI) models utilizing whole slide image (WSI) in patients with T1 CRC. In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, a systematic review was conducted through searches in PubMed (MEDLINE), Embase, and the Cochrane Library for relevant studies published up to December 2023. The inclusion criteria were studies assessing the accuracy of hematoxylin and eosin-stained WSI-based AI models for predicting LNM in patients with T1 CRC. Four studies met the criteria for inclusion in this systematic review. The area under the receiver operating characteristic curve for these AI models ranged from 0.57 to 0.76. In the three studies in which AI performance was compared directly with current treatment guidelines, AI consistently exhibited a higher area under the receiver operating characteristic curve. At a fixed sensitivity of 100%, specificities ranged from 18.4% to 45.0%. Artificial intelligence models based on WSI can potentially address the issue of diagnostic variability between pathologists and exceed the predictive accuracy of current guidelines. However, these findings require confirmation by larger studies that incorporate external validation.

Development and validation of a deep learning pipeline to diagnose ovarian masses using ultrasound screening: a retrospective multicenter study

Ovarian cancer has the highest mortality rate among gynaecological malignancies and is initially screened using ultrasound. Owing to the high complexity of ultrasound images of ovarian masses and the anatomical characteristics of the deep pelvic cavity, subjective assessment requires extensive experience and skill. Therefore, detecting the ovaries and ovarian masses and diagnose ovarian cancer are challenging. In the present study, we aimed to develop an automated deep learning framework, the Ovarian Multi-Task Attention Network (OvaMTA), for ovary and ovarian mass detection, segmentation, and classification, as well as further diagnosis of ovarian masses based on ultrasound screening. Between June 2020 and May 2022, the OvaMTA model was trained, validated and tested on a training and validation cohort including 6938 images and an internal testing cohort including 1584 images which were recruited from 21 hospitals involving women who underwent ultrasound examinations for ovarian masses. Subsequently, we recruited two external test cohorts from another two hospitals. We obtained 1896 images between February 2024 and April 2024 as image-based external test dataset, and further obtained 159 videos for the video-based external test dataset between April 2024 and May 2024. We developed an artificial intelligence (AI) system (termed OvaMTA) to diagnose ovarian masses using ultrasound screening. It includes two models: an entire image-based segmentation model, OvaMTA-Seg, for ovary detection and a diagnosis model, OvaMTA-Diagnosis, for predicting the pathological type of ovarian mass using image patches cropped by OvaMTA-Seg. The performance of the system was evaluated in one internal and two external validation cohorts, and compared with doctors' assessments in real-world testing. We recruited eight physicians to assess the real-world data. The value of the system in assisting doctors with diagnosis was also evaluated. In terms of segmentation, OvaMTA-Seg achieved an average Dice score of 0.887 on the internal test set and 0.819 on the image-based external test set. OvaMTA-Seg also performed well in ovarian mass detection from test images, including healthy ovaries and masses (internal test area under the curve [AUC]: 0.970; external test AUC: 0.877). In terms of classification diagnosis prediction, OvaMTA-Diagnosis demonstrated high performance on image-based internal (AUC: 0.941) and external test sets (AUC: 0.941). In video-based external testing, OvaMTA recognised 159 videos with ovarian masses with AUC of 0.911, and is comparable to the performance of senior radiologists (ACC: 86.2 vs. 88.1, p=0.50; SEN: 81.8 vs. 88.6, p=0.16; SPE: 89.2 vs. 87.6, p=0.68). There was a significant improvement in junior and intermediate radiologists who were assisted by AI compared to those who were not assisted by AI (ACC: 80.8 vs. 75.3, p=0.00015; SEN: 79.5 vs. 74.6, p=0.029; SPE: 81.7 vs. 75.8, p=0.0032). General practitioners assisted by AI achieved an average performance of radiologists (ACC: 82.7 vs. 81.8, p=0.80; SEN: 84.8 vs. 82.6, p=0.72; SPE: 81.2 vs. 81.2, p>0.99). The OvaMTA system based on ultrasound imaging is a simple and practical auxiliary tool for screening for ovarian cancer, with a diagnostic performance comparable to that of senior radiologists. This provides a potential tool for screening ovarian cancer. This work was supported by the National Natural Science Foundation of China (Grant Nos. 12090020, 82071929, and 12090025) and the R&D project of the Pazhou Lab (Huangpu) (Grant No. 2023K0605).

Yield prediction through UAV-based multispectral imaging and deep learning in rice breeding trials

ContextPredicting crop yields with high precision and timeliness is essential for crop breeding, enabling the optimization of planting strategies and efficients resource allocation while ensuring food security. Current research in this field typically does not address the problem of yield prediction in the diverse context of breeding experiments involving numerous varieties. However, evaluating the performance of prediction models across multiple varieties is vital for further model refining and enhancing model robustness and adaptability. ObjectiveThis study aims to evaluate the performance of feature- and image-based yield prediction models for yields with multiple varieties to compare their capabilities and determine an appropriate timing for early yield prediction. MethodsThis study combines unmanned aerial vehicle (UAV)-based multispectral remote sensing imagery with machine learning and deep learning-based algorithms to develop rice yield prediction models across multiple varieties. The performances of both feature- and image-based models are evaluated. The feature-based models considered in this study include random forest (RF), deep neural network (DNN), and long short-term memory (LSTM) algorithms, and the image-based models are convolutional neural network (CNN) architectures, including both two-dimensional (2D) and three-dimensional (3D) CNN models. To assess the performance of the multi-variety crop yield prediction models thoroughly, this study considers two sampling scenarios: stratified sampling and group sampling. Results and conclusionsThe results show that the image-based deep learning models outperform the feature-based machine learning models, which indicates their superior robustness in multi-variety scenarios and highlights their significant potential of directly extracting spatiotemporal features from images for yield prediction. The results indicate that the multi-temporal 2D CNN model (i.e., the CNN-M2D model) can achieve the best yield prediction performance among all models, achieving RRMSE = 8.13 % and R2 = 0.73. The prediction results also demonstrate good consistency with the observed data, indicating an efficient capturing of spatial pattern variations in yield across different varieties. Based on the results, with the crops progressing along the growth stages, the accuracy of the yield prediction models improves gradually, achieving the best prediction performance during the flowering to grain-filling stage. Finally, according to the results, the optimal lead time for predicting rice yield is approximately one month before harvest. SignificanceOur study can provide a reference for the research community in yield prediction and high-yield variety selection in breeding trials.

Development of a CT image-based virtual atelectasis simulation model and noninvasive lung nodule localization system.

In the process of video-assisted thoracoscopic surgery (VATS) for lung nodule resection, lung is leaded to atelectasis. However, preoperative computed tomography (CT) images are taken during inspiration, which means they differ significantly from the lung status observed during surgery. Consequently, this discrepancy can make the localization of small or subsolid nodules challenging during the operation. This study aimed to develop a CT-based virtual atelectasis simulation system for noninvasive lung nodule localization. By accurately simulating atelectasis, this study aimed to improve the precision of presurgical planning from lung nodule resections. This study retrospectively examined 20 patients who had either subsolid nodules or small nodules less than 3 cm in size, selected from a cohort of 279 patients who underwent VATS surgery for lung nodules in Korea University Ansan Hospital between June 28, 2021, and January 22, 2024. Chest CT images of the lungs of 20 patients were acquired, and image data were converted three-dimensional models. The mesh points extracted from these lung models were manipulated to simulate the effects of gravity, by adjusting the lung shapes and nodule locations to align with the respective surgical postures of the patients. Subsequently, we assessed the similarity of the simulation by comparing the resulting deformed lung shapes and nodule locations with the corresponding perspectives observed in the surgical videos. The average volume of the entire lung among the patients was 2,336 cm3 (±588). After atelectasis simulation, the average lung shrinkage rate was 48.6% (±12.9%). Evaluations of an average of 15 pairs of images per case revealed significant conformity between atelectasis simulation images and surgical video snapshots, with average Dice and Jaccard similarity coefficient values of 90.27 and 88.25, respectively. Furthermore, the alignment of nodule locations between the simulations and surgical anticipation demonstrated notable accuracy, with an average Hausdorff distance of 6.39 mm. We successfully developed a simulation of lung atelectasis based on preoperative CT scans that closely resembled actual surgical videos. The integration of this presurgical atelectasis simulation is anticipated to enhance the accuracy of nodule locations, thus contributing to more efficient and precise surgical planning.

Adaptive CLIP for open-domain 3D model retrieval

In order to effectively enhance the practicality of 3D model retrieval, we adopt a single real image as the query sample for retrieving 3D models. However, the significant differences between 2D images and 3D models in terms of lighting conditions, textures and backgrounds, posing a great challenge for accurate retrieval. Existing work on 3D model retrieval mainly focuses on closed-domain research, while the open-domain condition where the category relationship between the query image and the 3D model is unknown is more in line with the needs of real scenarios. CLIP shows significant promise in comprehending open-world visual concepts, facilitating effective zero-shot image recognition. Based on this multimodal pre-training large language model, we introduce Adaptive Open-domain Semantic Nearest-neighbor Contrast (AOSNC), a method for learning and aligning multi-modal text, image, and 3D model. In order to solve the issue of inconsistent cross-domain categories and difficult sample correlation in open-domain, we construct a cross-modal bridge using CLIP. This model utilizes textual features to bridge the gap between 2D images and 3D model views. Additionally, we design an adaptive network layer to address the limitations of the pre-training model for 3D model views and enhance cross-modal alignment. We propose a mutual nearest-neighbor semantic alignment loss to address the challenge of aligning features from disparate modalities (text, images, and 3D models). This loss function enhances cross-modal learning by effectively associating and distinguishing features, improving retrieval accuracy. We conducted comprehensive experiments using the image-based 3D model retrieval dataset MI3DOR and the cross-domain 3D model retrieval dataset NTU-PSB to validate the superiority of the proposed method. Our results show significant improvements in several evaluation metrics, underscoring the efficacy of our method in augmenting cross-modal feature alignment and retrieval performance.