Towards Improving Bio-Image Segmentation Quality Through Ensemble Post-processing of Deep Learning and Classical 3D Segmentation Pipelines

Segmenting three-dimensional (3D) microscopy images is essential for understanding phenomena like morphogenesis, cell division, cellular growth, and genetic expression patterns. Recently, deep learning (DL) pipelines have been developed, which claim to provide high accuracy segmentation of cellular images and are increasingly considered as the state of the art for image segmentation problems. However, it remains difficult to define their relative performances as the concurrent diversity and lack of uniform evaluation strategies makes it difficult to know how their results compare. In this paper, we first made an inventory of the available DL methods for 3D cell segmentation. We next implemented and quantitatively compared a number of representative DL pipelines, alongside a highly efficient non-DL method named MARS. The DL methods were trained on a common dataset of 3D cellular confocal microscopy images. Their segmentation accuracies were also tested in the presence of different image artifacts. A specific method for segmentation quality evaluation was adopted, which isolates segmentation errors due to under- or oversegmentation. This is complemented with a 3D visualization strategy for interactive exploration of segmentation quality. Our analysis shows that the DL pipelines have different levels of accuracy. Two of them, which are end-to-end 3D and were originally designed for cell boundary detection, show high performance and offer clear advantages in terms of adaptability to new data.

Bi-stage multi-modal 3D instance segmentation method for production workshop scene

3D object recognition is a challenging task for intelligent and robot systems\nin industrial and home indoor environments. It is critical for such systems to\nrecognize and segment the 3D object instances that they encounter on a frequent\nbasis. The computer vision, graphics, and machine learning fields have all\ngiven it a lot of attention. Traditionally, 3D segmentation was done with\nhand-crafted features and designed approaches that did not achieve acceptable\nperformance and could not be generalized to large-scale data. Deep learning\napproaches have lately become the preferred method for 3D segmentation\nchallenges by their great success in 2D computer vision. However, the task of\ninstance segmentation is currently less explored. In this paper, we propose a\nnovel approach for efficient 3D instance segmentation using red green blue and\ndepth (RGB-D) data based on deep learning. The 2D region based convolutional\nneural networks (Mask R-CNN) deep learning model with point based rending\nmodule is adapted to integrate with depth information to recognize and segment\n3D instances of objects. In order to generate 3D point cloud coordinates (x, y,\nz), segmented 2D pixels (u, v) of recognized object regions in the RGB image\nare merged into (u, v) points of the depth image. Moreover, we conducted an\nexperiment and analysis to compare our proposed method from various points of\nview and distances. The experimentation shows the proposed 3D object\nrecognition and instance segmentation are sufficiently beneficial to support\nobject handling in robotic and intelligent systems.\n

Advancements in artificial intelligence for biometrics: A deep dive into model-based gait recognition techniques

Deep learning-based quantitative assessment of renal chronicity indices in lupus nephritis.

Real-time gait biometrics for surveillance applications: A review

BackgroundTo investigate the ability of artificial intelligence (AI)-based and semi-quantitative dynamic contrast enhanced (DCE) multiparametric MRI (mpMRI), performed within [18F]-PSMA-1007 PET/MRI, in differentiating benign from malignant prostate tissues in patients with primary prostate cancer (PC).ResultsA total of seven patients underwent whole-body [18F]-PSMA-1007 PET/MRI examinations including a pelvic mpMRI protocol with T2w, diffusion weighted imaging (DWI) and DCE image series. Conventional analysis included visual reading of PET/MRI images and Prostate Imaging Reporting & Data System (PI-RADS) scoring of the prostate. On the prostate level, we performed manual segmentations for time-intensity curve parameter formation and semi-quantitative analysis based on DCE segmentation data of PC-suspicious lesions. Moreover, we applied a recently introduced deep learning (DL) pipeline previously trained on 1010 independent MRI examinations with systematic biopsy-enhanced histopathological targeted biopsy lesion ground truth in order to perform AI-based lesion detection, prostate segmentation and derivation of a deep learning PI-RADS score. DICE coefficients between manual and automatic DL-acquired segmentations were compared. On patient-based analysis, PET/MRI revealed PC-suspicious lesions in the prostate gland in 6/7 patients (Gleason Score-GS ≥ 7b) that were histologically confirmed. Four of these patients also showed lymph node metastases, while two of them had bone metastases. One patient with GS 6 showed no PC-suspicious lesions. Based on DCE segmentations, a distinction between PC-suspicious and normal appearing tissue was feasible with the parameters fitted maximum contrast ratio (FMCR) and wash-in-slope. DICE coefficients (manual vs. deep learning) were comparable with literature values at a mean of 0.44. Further, the DL pipeline could identify the intraprostatic PC-suspicious lesions in all six patients with clinically significant PC.ConclusionFirstly, semi-quantitative DCE analysis based on manual segmentations of time-intensity curves was able to distinguish benign from malignant tissues. Moreover, DL analysis of the MRI data could detect clinically significant PC in all cases, demonstrating the feasibility of AI-supported approaches in increasing diagnostic certainty of PSMA-radioligand PET/MRI.

Complete pulmonary fissures are assessed on computed tomography (CT) and are required for emphysema patients to be successfully treated with endobronchial valve (EBV) therapy. We propose a deep learning (DL) pipeline that uses a patch-based approach to quantitatively assess fissure completeness on CT and evaluate it in a clinical trial cohort. From the EBV for emphysema palliation trial (VENT), 130 CT scans were used in this study. The DL model utilizes nnU-Net as a backbone for the automatic pre- and post-processing of CT images and configuration of a 3D U-Net to segment patches of fissure and non-fissure. Five-fold cross validation is applied for training and inferences are obtained using a sliding window approach. Average symmetric surface distance (ASSD) and surface dice coefficient (SDC) at a threshold of 2mm evaluates segmentation performance. A fissure integrity score (FIS) is calculated as the percentage of complete fissure voxels along the surface of the assumed interlobar region using pulmonary lobar segmentations. A predicted-FIS (p-FIS) is quantified from the CNN output and is compared to the reference-FIS (r-FIS) as complete (FIS≥90%), partial (10%≤ FIS< 90%) or absent (FIS< 10%). A mean(±SD) SDC of 0.95(±0.037) is achieved for the left oblique fissure (LOF); 0.84(±0.144) for the right horizontal fissure (RHF), and 0.94(±0.098) for the right oblique fissure (ROF). Concordance rate of p-FIS and r-FIS is 86.4%, 88.6%, and 86.4% for the LOF, RHF, and ROF, respectively. A DL pipeline using a patch-based approach has potential to segment interlobar fissures from CT to quantitatively assess fissure completeness.

A robust transformer-based pipeline of 3D cell alignment, denoise and instance segmentation on electron microscopy sequence images

BACKGROUND. Skeletal muscle area (SMA), representing skeletal muscle cross-sectional area at the L3 vertebral level, and skeletal muscle index (SMI), representing height-normalized SMA, can serve as markers of sarcopenia. Normal SMA and SMI values have been reported primarily in adults. OBJECTIVE. The purpose of this study was to use an automated deep learning (DL) pipeline for muscle segmentation on abdominal CT to define normative age- and sex-based values for pediatric muscle cross-sectional area as a guide for diagnosis of sarcopenia in children. METHODS. This retrospective study reviewed records of patients who underwent abdominal CT at Cincinnati Children's Hospital Medical Center from January 1, 2009, to January 3, 2019. Patients were excluded on the basis of age outside of the eligible range (2.00-18.99 years), body mass index (BMI) outside of 5-95% age-based percentiles using CDC and WHO growth charts, known medical condition, medication use, support devices, surgery, or missing axial images at the L3 level. A previously validated automated DL pipeline was used to identify an axial slice at L3 and segment skeletal muscle to generate SMA and SMI. Pearson correlation coefficients were computed. Quantile regression analysis was used to plot SMA and SMI as functions of age and sex and to determine age- and sex-based percentile values. RESULTS. Of 8817 patients who underwent abdominal CT during the study period, 2168 (mean age, 12.3 ± 4.3 [SD] years; 1125 female patients, 1043 male patients) met inclusion criteria. Mean BMI-for-age percentile based on CDC and WHO growth charts was 64.8% ± 25.3% for female patients and 61.4% ± 25.8% for male patients. SMA showed strong correlation with weight, height, age, and BMI for male (0.79-0.94) and female (0.75-0.90) patients; SMI showed weak-to-moderate correlation with weight, height, age, and BMI for male (0.25-0.57) and female (0-0.43) patients. Normal SMA and SMI ranges for age and sex were expressed as curves and as a lookup table, identifying 54 male and 59 female patients with muscle measurements below the 5th percentile regression curve. CONCLUSION. By using an automated DL pipeline in a large sample of carefully selected children, normal ranges for SMA and SMI were calculated as functions of age and sex. CLINICAL IMPACT. The normative values should aid the diagnosis of sarcopenia in children.

Prognostic models for spinal cord astrocytoma patients are lacking due to the low incidence of the disease. Here, we aim to develop a fully automated deep learning (DL) pipeline for stratified overall survival (OS) prediction based on preoperative MR images. A total of 587 patients diagnosed with intramedullary tumors were retrospectively enrolled in our hospital to develop an automated pipeline for tumor segmentation and OS prediction. The automated pipeline included a T2WI-based tumor segmentation model and 3 cascaded binary OS prediction models (1-year, 3-year, and 5-year models). For the tumor segmentation model, 439 cases of intramedullary tumors were used to model training and testing using a transfer learning strategy. A total of 138 patients diagnosed with astrocytomas were included to train and test the OS prediction models via 10 × 10-fold cross-validation using CNNs. The dice of the tumor segmentation model with the test set was 0.852. The results indicated that the best input of OS prediction models was a combination of T2W and T1C images and the tumor mask. The 1-year, 3-year, and 5-year automated OS prediction models achieved accuracies of 86.0%, 84.0%, and 88.0% and AUCs of 0.881 (95% CI 0.839-0.918), 0.862 (95% CI 0.827-0.901), and 0.905 (95% CI 0.867-0.942), respectively. The automated DL pipeline achieved 4-class OS prediction (<1 year, 1-3 years, 3-5 years, and >5 years) with 75.3% accuracy. We proposed an automated DL pipeline for segmenting spinal cord astrocytomas and stratifying OS based on preoperative MR images.

BackgroundVariability in the bony morphology of pathologic hips/knees is a challenge in automating preoperative computed tomography (CT) scan measurements. With the increasing prevalence of CT for advanced preoperative planning, processing this data represents a critical bottleneck in presurgical planning, research, and development. The purpose of this study was to demonstrate a reproducible and scalable methodology for analyzing CT-based anatomy to process hip and knee anatomy for perioperative planning and execution. MethodsOne hundred patients with preoperative CT scans undergoing total knee arthroplasty for osteoarthritis were processed. A two-step deep learning pipeline of classification and segmentation models was developed that identifies landmark images and then generates contour representations. We utilized an open-source computer vision library to compute measurements. Classification models were assessed by accuracy, precision, and recall. Segmentation models were evaluated using dice and mean Intersection over Union (IOU) metrics. Contour measurements were compared against manual measurements to validate posterior condylar axis angle, sulcus angle, trochlear groove-tibial tuberosity distance, acetabular anteversion, and femoral version. ResultsClassifiers identified landmark images with accuracy of 0.91 and 0.88 for hip and knee models, respectively. Segmentation models demonstrated mean IOU scores above 0.95 with the highest dice coefficient of 0.957 [0.954-0.961] (UNet3+) and the highest mean IOU of 0.965 [0.961-0.969] (Attention U-Net). There were no statistically significant differences for the measurements taken automatically vs manually (P > 0.05). Average time for the pipeline to preprocess (48.65 +/− 4.41 sec), classify/retrieve landmark images (8.36 +/− 3.40 sec), segment images (<1 sec), and obtain measurements was 2.58 (+/− 1.92) minutes. ConclusionsA fully automated three-stage deep learning and computer vision-based pipeline of classification and segmentation models accurately localized, segmented, and measured landmark hip and knee images for patients undergoing total knee arthroplasty. Incorporation of clinical parameters, like patient-reported outcome measures and instability risk, will be important considerations alongside anatomic parameters.

Instance segmentation and object detection are significant problems in the fields of computer vision and robotics. We address those problems by proposing a novel object segmentation and detection system. First, we detect 2D objects based on RGB, depth only, or RGB-D images. A 3D convolutional-based system, named Frustum VoxNet, is proposed. This system generates frustums from 2D detection results, proposes 3D candidate voxelized images for each frustum, and uses a 3D convolutional neural network (CNN) based on these candidates voxelized images to perform the 3D instance segmentation and object detection. Results on the SUN RGB-D dataset show that our RGB-D-based system’s 3D inference is much faster than state-of-the-art methods, without a significant loss of accuracy. At the same time, we can provide segmentation and detection results using depth only images, with accuracy comparable to RGB-D-based systems. This is important since our methods can also work well in low lighting conditions, or with sensors that do not acquire RGB images. Finally, the use of segmentation as part of our pipeline increases detection accuracy, while providing at the same time 3D instance segmentation.

Independently exploring unknown spaces or finding objects in an indoor environment is a daily but challenging task for visually impaired people. However, common 2D assistive systems lack depth relationships between various objects, resulting in difficulty to obtain accurate spatial layout and relative positions of objects. To tackle these issues, we propose HIDA, a lightweight assistive system based on 3D point cloud instance segmentation with a solid-state LiDAR sensor, for holistic indoor detection and avoidance. Our entire system consists of three hardware components, two interactive functions (obstacle avoidance and object finding) and a voice user interface. Based on voice guidance, the point cloud from the most recent state of the changing indoor environment is captured through an on-site scanning performed by the user. In addition, we design a point cloud segmentation model with dual lightweight decoders for semantic and offset predictions, which satisfies the efficiency of the whole system. After the 3D instance segmentation, we post-process the segmented point cloud by removing outliers and projecting all points onto a top-view 2D map representation. The system integrates the information above and interacts with users intuitively by acoustic feedback. The proposed 3D instance segmentation model has achieved state-of-the-art performance on ScanNet v2 dataset. Comprehensive field tests with various tasks in a user study verify the usability and effectiveness of our system for assisting visually impaired people in holistic indoor understanding, obstacle avoidance and object search.

While most recent work in room instance segmentation relies on orthographic top-down projections of 3D point clouds to 2D density maps, leading to information loss of one dimension, 3D instance segmentation methods based on deep learning were rarely considered. We explore the potential of the general 3D instance segmentation deep learning model Mask3D for room instance segmentation in indoor building point clouds. We show that Mask3D generates meaningful predictions for multi-floor scenes. After hyperparameter optimization, Mask3D outperforms the current state-of-the-art method RoomFormer evaluated in 3D on the synthetic Structured3D dataset. We provide generalization results of Mask3D trained on Structured3D to the real-world S3DIS and Matterport3D datasets, showing a domain gap. Fine-tuning improves the results. In contrast to related work in room instance segmentation, we employ the more expressive mean average precision (mAP) metric, and we propose the more intuitive successfully detected rooms (SDR) metric, which is an absolute recall measure. Our results indicate potential for the digitization of the construction industry.