Ground Truth Annotations Research Articles

Texture appearance model, a new model-based segmentation paradigm, application on the segmentation of lung nodule in the CT scan of the chest

Lung cancer causes more than one million deaths worldwide each year. Averages of 5-year survival rate of patients with Non-small cell lung cancer (NSCLC), which is the most common type of lung cancer, is 15%. Computer-Aided Detection (CAD) is a very important tool for identifying lung lesions in medical imaging. In general, the process line of a CAD system can be divided into four main stages: preprocessing, localization, feature extraction, and classification. As segmentation is required for localization in computer vision and medical image analysis, this step has become a major and challenging problem, and much research has been done on new segmentation techniques. In recent years, interest in model-based segmentation methods has increased, and the reason for this is even if some object information is lost, such gaps can be filled by using the previous information in the model. This paper proposed Texture Appearance Model (TAM), which is a new model-based method and segments all types of nodule areas accurately and efficiently, including juxta-pleural nodules, without separating the lung from the surrounding area in a CT scan of the lung. In this method, Texture Representation of Image (TRI) is obtained using tissue texture feature extraction and feature selection algorithms. The proposed method has been evaluated in 85 nodules of the dataset, received from the Iranian hospital, in which the ground-truth annotation by physicians and CT imaging data were provided. The results show that the proposed algorithm has an encouraging performance for distinguishing different types of nodules, including pleural, cavity and non-solid nodules, achieving an average dice similarity coefficient (DSC) of 84.75%.

Automatic detection of attachment sites for knee ligaments and tendons on CT images.

The diseases and injuries of the knee joint are the most common orthopedic disorders. Personalized knee models can be helpful in the process of early intervention and lasting treatment techniques development. Fully automatic reconstruction of knee joint anatomical structures from medical images (CT, MRI, ultrasound) remains a challenge. For this reason, most of state-of-the-art knee joint models contain simplifications such as representation of muscles and ligaments as line segments connecting two points which replace attachment areas. The paper presents algorithms for automatic detection of such points on knee CT images. This paper presents three approaches to automatic detection of ligaments and tendons attachment sites on the patients CT images: qualitative anatomical descriptions, analysis of bones curvature, and quantitative anatomical descriptions. Combinations of these approaches result in new automatic detection algorithms. Each algorithm exploits anatomical peculiarities of each attachment site, e.g., bone curvature and number of other attachments in a neighborhood of the site. The experimental dataset consisted of 26 anonymized CT sequences containing right and left knee joints in different resolutions. The proposed algorithms take into account bone surface curvatures and spatial differences in locations of medial and lateral parts of both knees. The algorithms for detection of quadriceps femoris, popliteus, biceps femoris tendons, and lateral collateral and medial collateral ligaments attachment sites are provided, as well as examples of their application. Two algorithms are validated by comparison with known statistics of ligaments lengths and also using ground truth annotations for anatomical landmarks approved by clinical experts. The algorithms simplify generation of patient-specific knee joint models demanded in personalized biomechanical models. The algorithms in the current implementation have two important limitations. First, the correctness of the produced results depends on the bones segmentation quality. Second, the presented algorithms detect a point of the attachment site, which is not necessarily its center. Therefore, manual correction of the attachment site location may be required for attachments with relatively large area.

Inter-Variability Study of COVLIAS 1.0: Hybrid Deep Learning Models for COVID-19 Lung Segmentation in Computed Tomography.

Background: For COVID-19 lung severity, segmentation of lungs on computed tomography (CT) is the first crucial step. Current deep learning (DL)-based Artificial Intelligence (AI) models have a bias in the training stage of segmentation because only one set of ground truth (GT) annotations are evaluated. We propose a robust and stable inter-variability analysis of CT lung segmentation in COVID-19 to avoid the effect of bias. Methodology: The proposed inter-variability study consists of two GT tracers for lung segmentation on chest CT. Three AI models, PSP Net, VGG-SegNet, and ResNet-SegNet, were trained using GT annotations. We hypothesized that if AI models are trained on the GT tracings from multiple experience levels, and if the AI performance on the test data between these AI models is within the 5% range, one can consider such an AI model robust and unbiased. The K5 protocol (training to testing: 80%:20%) was adapted. Ten kinds of metrics were used for performance evaluation. Results: The database consisted of 5000 CT chest images from 72 COVID-19-infected patients. By computing the coefficient of correlations (CC) between the output of the two AI models trained corresponding to the two GT tracers, computing their differences in their CC, and repeating the process for all three AI-models, we show the differences as 0%, 0.51%, and 2.04% (all < 5%), thereby validating the hypothesis. The performance was comparable; however, it had the following order: ResNet-SegNet > PSP Net > VGG-SegNet. Conclusions: The AI models were clinically robust and stable during the inter-variability analysis on the CT lung segmentation on COVID-19 patients.

Self-Supervised Feature Learning of 1D Convolutional Neural Networks with Contrastive Loss for Eating Detection Using an In-Ear Microphone.

The importance of automated and objective monitoring of dietary behavior is becoming increasingly accepted. The advancements in sensor technology along with recent achievements in machine-learning-based signal-processing algorithms have enabled the development of dietary monitoring solutions that yield highly accurate results. A common bottleneck for developing and training machine learning algorithms is obtaining labeled data for training supervised algorithms, and in particular ground truth annotations. Manual ground truth annotation is laborious, cumbersome, can sometimes introduce errors, and is sometimes impossible in free-living data collection. As a result, there is a need to decrease the labeled data required for training. Additionally, unlabeled data, gathered in-the-wild from existing wearables (such as Bluetooth earbuds) can be used to train and fine-tune eating-detection models. In this work, we focus on training a feature extractor for audio signals captured by an in-ear microphone for the task of eating detection in a self-supervised way. We base our approach on the SimCLR method for image classification, proposed by Chen et al. from the domain of computer vision. Results are promising as our self-supervised method achieves similar results to supervised training alternatives, and its overall effectiveness is comparable to current state-of-the-art methods. Code is available at https://github.com/mug-auth/ssl-chewing.

Calibration and three-dimensional reconstruction with a photorealistic simulator based on the omnidirectional vision system

Recent advancements in deep learning require a large amount of the annotated training data containing various terms and conditions of the environment. Thus, developing and testing algorithms for the navigation of mobile robots can be expensive and time-consuming. Motivated by the aforementioned problems, this article presents a photorealistic simulator for the computer vision community working with omnidirectional vision systems. Built using unity, the simulator integrates sensors, mobile robots, and elements of the indoor environment and allows one to generate synthetic photorealistic data sets with automatic ground truth annotations. With the aid of the proposed simulator, two practical applications are studied, namely extrinsic calibration of the vision system and three-dimensional reconstruction of the indoor environment. For the proposed calibration and reconstruction techniques, the processes themselves are simple, robust, and accurate. Proposed methods are evaluated experimentally with data generated by the simulator. The proposed simulator and supporting materials are available online: http://www.ilabit.org .

Automated vessel segmentation in lung CT and CTA images via deep neural networks

The distribution of pulmonary vessels in computed tomography (CT) and computed tomography angiography (CTA) images of lung is important for diagnosing disease, formulating surgical plans and pulmonary research. Based on the pulmonary vascular segmentation task of International Symposium on Image Computing and Digital Medicine 2020 challenge, this paper reviews 12 different pulmonary vascular segmentation algorithms of lung CT and CTA images and then objectively evaluates and compares their performances. First, we present the annotated reference dataset of lung CT and CTA images. A subset of the dataset consisting 7,307 slices for training and 3,888 slices for testing was made available for participants. Second, by analyzing the performance comparison of different convolutional neural networks from 12 different institutions for pulmonary vascular segmentation, the reasons for some defects and improvements are summarized. The models are mainly based on U-Net, Attention, GAN, and multi-scale fusion network. The performance is measured in terms of Dice coefficient, over segmentation rate and under segmentation rate. Finally, we discuss several proposed methods to improve the pulmonary vessel segmentation results using deep neural networks. By comparing with the annotated ground truth from both lung CT and CTA images, most of 12 deep neural network algorithms do an admirable job in pulmonary vascular extraction and segmentation with the dice coefficients ranging from 0.70 to 0.85. The dice coefficients for the top three algorithms are about 0.80. Study results show that integrating methods that consider spatial information, fuse multi-scale feature map, or have an excellent post-processing to deep neural network training and optimization process are significant for further improving the accuracy of pulmonary vascular segmentation.

Automated chart filing on panoramic radiographs using deep learning

ObjectiveThe aim of this study is to automatically detect, segment and label teeth, crowns, fillings, root canal fillings, implants and root remnants on panoramic radiographs (PR(s)). Material and methodsAs a reference, 2000 PR(s) were manually annotated and labeled. A deep-learning approach based on mask R-CNN with Resnet-50 in combination with a rule-based heuristic algorithm and a combinatorial search algorithm was trained and validated on 1800 PR(s). Subsquently, the trained algorithm was applied onto a test set consisting of 200 PR(s). F1 scores, as a measure of accuracy, were calculated to quantify the degree of similarity between the annotated ground-truth and the model predictions. The F1-score considers the harmonic mean of precison (positive predictive value) and recall (specificity). ResultsThe proposes method achieved F1 scores up to 0.993, 0.952 and 0.97 for detection, segmentation and labeling, respectivley. ConclusionThe proposed method forms a promising foundation for the further development of automatic chart filing on PR(s). Clinical significanceDeep learning may assist clinicians in summarizing the radiological findings on panoramic radiographs. The impact of using such models in clinical practice should be explored.

Clinical Validation of Deep Learning Algorithms for Lung Cancer Radiotherapy Targeting

<h3>Purpose/Objective(s)</h3> Automated target segmentation for non-small cell lung cancer (NSCLC) patients has the potential to support radiation treatment planning. Artificial intelligence (AI) has demonstrated great promise in medical image segmentation tasks. However, most studies have been confined to <i>in silico</i> validation in small internal cohorts, lacking data on real-world clinical utility. In this study, we developed primary tumor and involved lymph node segmentation algorithms in computed tomography (CT) images. Validation is performed in multiple large multi-institutional cohorts to assess model generalizability. <h3>Materials/Methods</h3> Simulation CTs and ground truth annotations were collected from multiple public and private sources (total n = 2584). We employed the following benchmarks: Inter-observer (6 radiation oncologists, n = 20, median volumetric dice 0.83, 95% CI 0.82-0.84) and intra-observer (1 radiation oncologist, 3 reads, n = 21, median volumetric dice 0.88, 95% CI 0.84-0.9). We developed two segmentation algorithms: seed-point assisted and fully automated. Model training data (n = 787) comprised NSCLC-Radiomics (stages I-IIIB, n = 422) and LungRT-1 (stages IA-IV, n = 365). Validation was first performed in an internal dataset annotated by a single thoracic radiation oncologist (LungRT-1, n = 136). Additional validation included: (1) an internal dataset annotated by other radiation oncologists, including generalists, in our center (LungRT-2, n = 1075), (2) an external clinical trial dataset from 185 different institutions (RTOG-0617, n = 403), and (3) a dataset of early-stage surgical patients annotated for diagnostic purposes by radiologists (NSCLC-Radiogenomics, n = 142). Volumetric dice, using expert manual segmentations as ground truth, was used as an evaluation metric. <h3>Results</h3> The model performance is comparable to the benchmarks when validated on internal data, with degrading performance in cohorts annotated by other radiation oncologists. <h3>Conclusion</h3> The results highlight the importance of assessing segmentation style among annotators and understanding model generalizability in external cohorts, all while cautioning against degrading performance in increasingly external data. Differences between radiologists and radiation oncologists performing the same segmentation task underscore the importance of clinical context in AI model deployment. Further validation includes studying the dosimetric impact of AI-generated segmentations, and conducting human subject experiments to assess AI output acceptance and time savings.

Deep learning improves macromolecule identification in 3D cellular cryo-electron tomograms.

Cryogenic electron tomography (cryo-ET) visualizes the 3D spatial distribution of macromolecules at nanometer resolution inside native cells. However, automated identification of macromolecules inside cellular tomograms is challenged by noise and reconstruction artifacts, as well as the presence of many molecular species in the crowded volumes. Here, we present DeepFinder, a computational procedure that uses artificial neural networks to simultaneously localize multiple classes of macromolecules. Once trained, the inference stage of DeepFinder is faster than template matching and performs better than other competitive deep learning methods at identifying macromolecules of various sizes in both synthetic and experimental datasets. On cellular cryo-ET data, DeepFinder localized membrane-bound and cytosolic ribosomes (roughly 3.2 MDa), ribulose 1,5-bisphosphate carboxylase-oxygenase (roughly 560 kDa soluble complex) and photosystem II (roughly 550 kDa membrane complex) with an accuracy comparable to expert-supervised ground truth annotations. DeepFinder is therefore a promising algorithm for the semiautomated analysis of a wide range of molecular targets in cellular tomograms.

Fast and High-Quality, GPU-based, Deliberative, Object-Pose Estimation

Pose estimation of recognized objects is fundamental to tasks such as robotic grasping and manipulation. The need for reliable grasping imposes stringent accuracy requirements on pose estimation in cluttered, occluded scenes in dynamic environments. Modern methods employ large sets of training data to learn features and object templates in order to find correspondence between models and observed data. However, these methods require extensive annotation of ground-truth poses. An alternative is to use algorithms, such as PERCH (PErception Via SeaRCH) that seek an optimal explanation of the observed scene in a space of possible rendered versions. While PERCH offers strong guarantees on accuracy, the initial formulation suffers from poor scalability owing to its high runtime. In this work, we present PERCH 2.0, a deliberative approach that takes advantage of GPU acceleration and RGB data by formulating pose estimation as a single-shot, fully parallel approach. We show that PERCH 2.0 achieves a two orders of magnitude speedup (∼100X) over the hierarchical PERCH by evaluating thousands of poses in parallel. In addition, we propose a combined deliberative and discriminative framework for 6-DoF pose estimation that doesn’t require any ground-truth pose-annotation. Our work shows that PERCH 2.0 achieves, on the YCB-Video Dataset, a higher accuracy than DenseFusion, a state-of-the-art, end-to-end, learning-based approach. We also demonstrate that our work leads directly to an extension of deliberative pose estimation methods like PERCH to new domains, such as conveyor picking, which was previously infeasible due to high runtime. Our code is available at https://sbpl-cruz.github.io/perception/

Deep learning for automated detection and numbering of permanent teeth on panoramic images.

This study aimed to evaluate an automated detection system to detect and classify permanent teeth on orthopantomogram (OPG) images using convolutional neural networks (CNNs). In total, 591 digital OPGs were collected from patients older than 18 years. Three qualified dentists performed individual teeth labelling on images to generate the ground truth annotations. A three-step procedure, relying upon CNNs, was proposed for automated detection and classification of teeth. Firstly, U-Net, a type of CNN, performed preliminary segmentation of tooth regions or detecting regions of interest (ROIs) on panoramic images. Secondly, the Faster R-CNN, an advanced object detection architecture, identified each tooth within the ROI determined by the U-Net. Thirdly, VGG-16 architecture classified each tooth into 32 categories, and a tooth number was assigned. A total of 17,135 teeth cropped from 591 radiographs were used to train and validate the tooth detection and tooth numbering modules. 90% of OPG images were used for training, and the remaining 10% were used for validation. 10-folds cross-validation was performed for measuring the performance. The intersection over union (IoU), F1 score, precision, and recall (i.e. sensitivity) were used as metrics to evaluate the performance of resultant CNNs. The ROI detection module had an IoU of 0.70. The tooth detection module achieved a recall of 0.99 and a precision of 0.99. The tooth numbering module had a recall, precision and F1 score of 0.98. The resultant automated method achieved high performance for automated tooth detection and numbering from OPG images. Deep learning can be helpful in the automatic filing of dental charts in general dentistry and forensic medicine.

ADAADepth: Adapting Data Augmentation and Attention for Self-Supervised Monocular Depth Estimation

Self-supervised learning of depth has been a highly studied topic of research as it alleviates the requirement of having ground truth annotations for predicting depth. Depth is learnt as an intermediate solution to the task of view synthesis, utilising warped photometric consistency. Although it gives good results when trained using stereo data, the predicted depth is still sensitive to noise, illumination changes and specular reflections. Also, occlusion can be tackled better by learning depth from a single camera. We propose ADAA, utilising depth augmentation as depth supervision for learning accurate and robust depth. We propose a relational self-attention module that learns rich contextual features and further enhances depth results. We also optimize the auto-masking strategy across all losses by enforcing L1 regularisation over mask. Our novel progressive training strategy first learns depth at a lower resolution and then progresses to the original resolution with slight training. We utilise a ResNet18 encoder, learning features for prediction of both depth and pose. We evaluate our predicted depth on the standard KITTI driving dataset and achieve state-of-the-art results for monocular depth estimation whilst having significantly lower number of trainable parameters in our deep learning framework. We also evaluate our model on Make3D dataset showing better generalization than other methods.

Weakly-Supervised Single-view Dense 3D Point Cloud Reconstruction via Differentiable Renderer

In recent years, addressing ill-posed problems by leveraging prior knowledge contained in databases on learning techniques has gained much attention. In this paper, we focus on complete three-dimensional (3D) point cloud reconstruction based on a single red-green-blue (RGB) image, a task that cannot be approached using classical reconstruction techniques. For this purpose, we used an encoder-decoder framework to encode the RGB information in latent space, and to predict the 3D structure of the considered object from different viewpoints. The individual predictions are combined to yield a common representation that is used in a module combining camera pose estimation and rendering, thereby achieving differentiability with respect to imaging process and the camera pose, and optimization of the two-dimensional prediction error of novel viewpoints. Thus, our method allows end-to-end training and does not require supervision based on additional ground-truth (GT) mask annotations or ground-truth camera pose annotations. Our evaluation of synthetic and real-world data demonstrates the robustness of our approach to appearance changes and self-occlusions, through outperformance of current state-of-the-art methods in terms of accuracy, density, and model completeness.

A powerful image synthesis and semi-supervised learning pipeline for site-specific weed detection

Precise and efficient weed detection in agricultural fields is the key for robotic weed control. Recent developments in convolutional neural networks (CNNs) have achieved significant success in this regard. CNNs that simultaneously localize and classify objects in images are the predominant forms that have been widely used to detect crops and weeds. However, the use of CNNs in agriculture, particularly for weed detection, has been impeded by a lack of large training dataset with ground truth annotations. Cut-and-paste image synthesis approach and semi-supervised learning are popular methods to alleviate the training data deficiency problem. In this paper, we propose a novel image synthesis and semi-supervised learning pipeline to train site-specific weed detection models without the need for manually labeled images. The CNN models trained by this pipeline achieve performance levels close to that of the supervised models. We investigated the behavior of the proposed pipeline by varying several key components and showed that color match between the training and testing images, training-time color augmentation, and iterative semi-supervised learning largely improve the model performance. These promising results can be used to guide the construction of a weed image database applicable to different weed detection scenarios.

Assignment Flow for Order-Constrained OCT Segmentation

At the present time optical coherence tomography (OCT) is among the most commonly used non-invasive imaging methods for the acquisition of large volumetric scans of human retinal tissues and vasculature. The substantial increase of accessible highly resolved 3D samples at the optic nerve head and the macula is directly linked to medical advancements in early detection of eye diseases. To resolve decisive information from extracted OCT volumes and to make it applicable for further diagnostic analysis, the exact measurement of retinal layer thicknesses serves as an essential task be done for each patient separately. However, manual examination of OCT scans is a demanding and time consuming task, which is typically made difficult by the presence of tissue-dependent speckle noise. Therefore, the elaboration of automated segmentation models has become an important task in the field of medical image processing. We propose a novel, purely data driven geometric approach to order-constrained 3D OCT retinal cell layer segmentation which takes as input data in any metric space and can be implemented using only simple, highly parallelizable operations. As opposed to many established retinal layer segmentation methods, we use only locally extracted features as input and do not employ any global shape prior. The physiological order of retinal cell layers and membranes is achieved through the introduction of a smoothed energy term. This is combined with additional regularization of local smoothness to yield highly accurate 3D segmentations. The approach thereby systematically avoid bias pertaining to global shape and is hence suited for the detection of anatomical changes of retinal tissue structure. To demonstrate its robustness, we compare two different choices of features on a data set of manually annotated 3D OCT volumes of healthy human retina. The quality of computed segmentations is compared to the state of the art in automatic retinal layer segmention as well as to manually annotated ground truth data in terms of mean absolute error and Dice similarity coefficient. Visualizations of segmented volumes are also provided.

Feature‐fusion improves MRI single‐shot deep learning detection of small brain metastases

To examine whether feature-fusion (FF) method improves single-shot detector's (SSD's) detection of small brain metastases on contrast-enhanced (CE) T1-weighted MRI. The study included 234 MRI scansfrom 234 patients (64.3 years±12.0; 126 men). The ground-truth annotation was performed semiautomatically. SSDs with and without an FF module were developed and trained using 178 scans. The detection performance was evaluated at the SSDs' 50% confidence threshold using sensitivity, positive-predictive value (PPV), and the false-positive (FP) per scan with the remaining 56 scans. FF-SSD achieved an overall sensitivity of 86.0% (95% confidence interval [CI]: [83.0%, 85.6%]; 196/228) and 46.8% PPV (95% CI: [42.0%, 46.3%]; 196/434), with 4.3 FP (95% CI: [4.3, 4.9]). Lesions smaller than 3 mm had 45.8% sensitivity (95% CI: [36.1%, 45.5%]; 22/48) with 2.0 FP (95% CI: [1.9, 2.1]). Lesions measuring 3-6 mm had 92.3% sensitivity (95% CI: [86.5%, 92.0%]; 48/52) with 1.8 FP (95% CI: [1.7, 2.2]). Lesions larger than 6 mm had 98.4% sensitivity (95% CI: [97.8%, 99.4%]; 126/128) 0.5 FP (95% CI: [0.5, 0.8]) per scan. FF-SSD had a significantly higher sensitivity for lesions < 3 mm (p = 0.008, t = 3.53) than the baseline SSD, while the overall PPV was similar (p = 0.06, t = -2.16). A similar trend was observed even when the detector's confidence threshold was varied as low as 0.2, for which the FF-SSD's sensitivity was 91.2% and the FP was 9.5. The FF-SSD algorithm identified brain metastases on CE T1-weighted MRI with high accuracy.

Self-supervised multi-body scene flow estimation

In this paper, we address the problem of scene flow estimation from consecutive stereo pairs. In contrast to the state-of-the-art supervised learning based methods, we propose a self-supervised learning based pipeline that removes the requirement of large-scale ground truth annotations. Specifically, we employ a shared encoder StereoFlowNet to simultaneously learn optical flow estimation and disparity estimation, which not only achieves a compact network representation but also exploits the inherent connections between optical flow estimation and disparity estimation. To leverage the scene structure and motion representations, we propose to utilize a piece-wise planar model based disparity computation and multiple rigid body motion representation of the dynamic scene. In this way, the geometric and motion constraints play strong regularizations for the underlying problem. Experimental results on benchmarking dataset show that our proposed method achieves state-of-the-art performance in both optical flow and disparity estimation.

Interactive Machine Learning-Based Multi-Label Segmentation of Solid Tumors and Organs.

We seek the development and evaluation of a fast, accurate, and consistent method for general-purpose segmentation, based on interactive machine learning (IML). To validate our method, we identified retrospective cohorts of 20 brain, 50 breast, and 50 lung cancer patients, as well as 20 spleen scans, with corresponding ground truth annotations. Utilizing very brief user training annotations and the adaptive geodesic distance transform, an ensemble of SVMs is trained, providing a patient-specific model applied to the whole image. Two experts segmented each cohort twice with our method and twice manually. The IML method was faster than manual annotation by 53.1% on average. We found significant (p < 0.001) overlap difference for spleen (DiceIML/DiceManual = 0.91/0.87), breast tumors (DiceIML/DiceManual = 0.84/0.82), and lung nodules (DiceIML/DiceManual = 0.78/0.83). For intra-rater consistency, a significant (p = 0.003) difference was found for spleen (DiceIML/DiceManual = 0.91/0.89). For inter-rater consistency, significant (p < 0.045) differences were found for spleen (DiceIML/DiceManual = 0.91/0.87), breast (DiceIML/DiceManual = 0.86/0.81), lung (DiceIML/DiceManual = 0.85/0.89), the non-enhancing (DiceIML/DiceManual = 0.79/0.67) and the enhancing (DiceIML/DiceManual = 0.79/0.84) brain tumor sub-regions, which, in aggregation, favored our method. Quantitative evaluation for speed, spatial overlap, and consistency, reveals the benefits of our proposed method when compared with manual annotation, for several clinically relevant problems. We publicly release our implementation through CaPTk (Cancer Imaging Phenomics Toolkit) and as an MITK plugin.

Deep probabilistic tracking of particles in fluorescence microscopy images.

Tracking of particles in temporal fluorescence microscopy image sequences is of fundamental importance to quantify dynamic processes of intracellular structures as well as virus structures. We introduce a probabilistic deep learning approach for fluorescent particle tracking, which is based on a recurrent neural network that mimics classical Bayesian filtering. Compared to previous deep learning methods for particle tracking, our approach takes into account uncertainty, both aleatoric and epistemic uncertainty. Thus, information about the reliability of the computed trajectories is determined. Manual tuning of tracking parameters is not necessary and prior knowledge about the noise statistics is not required. Short and long-term temporal dependencies of individual object dynamics are exploited for state prediction, and assigned detections are used to update the predicted states. For correspondence finding, we introduce a neural network which computes assignment probabilities jointly across multiple detections as well as determines the probabilities of missing detections. Training requires only simulated data and therefore tedious manual annotation of ground truth is not needed. We performed a quantitative performance evaluation based on synthetic and real 2D as well as 3D fluorescence microscopy images. We used image data of the Particle Tracking Challenge as well as real time-lapse fluorescence microscopy images displaying virus structures and chromatin structures. It turned out that our approach yields state-of-the-art results or improves the tracking results compared to previous methods.

Artificial intelligence-based automated laparoscopic cholecystectomy surgical phase recognition and analysis

Artificial intelligence and computer vision have revolutionized laparoscopic surgical video analysis. However, there is no multi-center study focused on deep learning-based laparoscopic cholecystectomy phases recognizing. This work aims to apply artificial intelligence in recognizing and analyzing phases in laparoscopic cholecystectomy videos from multiple centers. This observational cohort-study included 163 laparoscopic cholecystectomy videos collected from four medical centers. Videos were labeled by surgeons and a deep-learning model was developed based on 90 videos. Thereafter, the performance of the model was tested in additional ten videos by comparing it with the annotated ground truth of the surgeon. Deep-learning models were trained to identify laparoscopic cholecystectomy phases. The performance of models was measured using precision, recall, F1 score, and overall accuracy. With a high overall accuracy of the model, additional 63 videos as an analysis set were analyzed by the model to identify different phases. Mean concordance correlation coefficient for annotations of the surgeons across all operative phases was 92.38%. Also, the overall phase recognition accuracy of laparoscopic cholecystectomy by the model was 91.05%. In the analysis set, there was an average surgery time of 2195 ± 896s, with a huge individual variance of different surgical phases. Notably, laparoscopic cholecystectomy in acute cholecystitis cases had prolonged overall durations, and the surgeon would spend more time in mobilizing the hepatocystic triangle phase. A deep-learning model based on multiple centers data can identify phases of laparoscopic cholecystectomy with a high degree of accuracy. With continued refinements, artificial intelligence could be utilized in huge data surgery analysis to achieve clinically relevant future applications.