Time-lapse Phase Contrast Microscopy Image Research Articles

Spatio-Temporal Mitosis Detection in Time-Lapse Phase-Contrast Microscopy Image Sequences: A Benchmark.

In this paper, we report the results of the first international contest on mitosis detection in phase-contrast microscopy image sequences (https://www.iti-tju.org/mitosisdetection), which was held at the workshop of computer vision for microscopy image analysis (CVMI) in CVPR 2019. This contest aims to promote research on spatiotemporal mitosis detection under microscopy images. In this contest, we released a large-scale time-lapse phase-contrast microscopy image dataset (C2C12-16) for the mitosis detection task. Compared with the previous popular datasets (e.g., C2C12, C3H10), C2C12-16 contains more annotated mitotic events and more diverse cell culture environments. A total of ten different mitosis detection methods were submitted in the contest and evaluated on the test sets of four different cell culture environments in C2C12-16. In this benchmark, we describe all methods and conduct a thorough analysis based on their performances and discuss a feasible direction for mitosis detection. To the best of our knowledge, this is the first benchmark for the mitosis detection problem using a time-lapse phase-contrast microscopy spatiotemporal image sequence model.

Deep Reinforcement Learning-Based Progressive Sequence Saliency Discovery Network for Mitosis Detection In Time-Lapse Phase-Contrast Microscopy Images.

Mitosis detection plays an important role in the analysis of cell status and behavior and is therefore widely utilized in many biological research and medical applications. In this article, we propose a deep reinforcement learning-based progressive sequence saliency discovery network (PSSD)for mitosis detection in time-lapse phase contrast microscopy images. By discovering the salient frames when cell state changes in the sequence, PSSD can more effectively model the mitosis process for mitosis detection. We formulate the discovery of salient frames as a Markov Decision Process (MDP)that progressively adjusts the selection positions of salient frames in the sequence, and further leverage deep reinforcement learning to learn the policy in the salient frame discovery process. The proposed method consists of two parts: 1)the saliency discovery module that selects the salient frames from the input cell image sequence by progressively adjusting the selection positions of salient frames; 2)the mitosis identification module that takes a sequence of salient frames and performs temporal information fusion for mitotic sequence classification. Since the policy network of the saliency discovery module is trained under the guidance of the mitosis identification module, PSSD can comprehensively explore the salient frames that are beneficial for mitosis detection. To our knowledge, this is the first work to implement deep reinforcement learning to the mitosis detection problem. In the experiment, we evaluate the proposed method on the largest mitosis detection dataset, C2C12-16. Experiment results show that compared with the state-of-the-arts, the proposed method can achieve significant improvement for both mitosis identification and temporal localization on C2C12-16.

Sequential Saliency Guided Deep Neural Network for Joint Mitosis Identification and Localization in Time-Lapse Phase Contrast Microscopy Images.

The analysis of cell mitotic behavior plays important role in many biomedical research and medical diagnostic applications. To improve the accuracy of mitosis detection in automated analysis systems, this paper proposes the sequential saliency guided deep neural network (SSG-DNN) to jointly identify and localize mitotic events in time-lapse phase contrast microscopy images. It consists of three key modules. First, the module of visual context learning extracts static visual feature and dynamic visual transition within individual volumetric cell regions. Secondly, with these information, the module of sequential saliency modeling aims to discover the saliency distribution over all successive frames in each volumetric region. Finally, the module of sequence structure modeling can leverage both visual context and saliency distribution for mitosis identification and localization. SSG-DNN can jointly realize visual feature learning and sequential structure modeling in the end-to-end framework. Moreover, the proposed method is independent of complicated preconditioning methods for mitotic candidate extraction and can be applied for mitosis detection in one-shot manner. To our knowledge, it is the first weakly supervised work to realize joint mitosis identification and localization only with sequence-wise labels. In our experiments, we evaluate its performances of both tasks on the popular C3H10 dataset and a novel and large-scale dataset, C2C12-16, which contains much more mitotic events and is more challenging owing to diverse cell culture conditions. Experimental results can demonstrate the superiority of the proposed method.

Cell mitosis event analysis in phase contrast microscopy images using deep learning.

In this paper, we solve the problem of mitosis event localization and its stage localization in time-lapse phase-contrast microscopy images. Our method contains three steps: first, we formulate a Low-Rank Matrix Recovery (LRMR) model to find salient regions from microscopy images and extract candidate patch sequences, which potentially contain mitosis events; second, we classify each candidate patch sequence by our proposed Hierarchical Convolution Neural Network (HCNN) with visual appearance and motion cues; third, for the detected mitosis sequences, we further segment them into four temporal stages by our proposed Two-stream Bidirectional Long-Short Term Memory (TS-BLSTM). In the experiments, we validate our system (LRMR, HCNN, and TS-BLSTM) and evaluate the mitosis event localization and stage localization performance. The proposed method outperforms state-of-the-arts by achieving 99.2% precision and 98.0% recall for mitosis event localization and 0.62 frame error on average for mitosis stage localization in five challenging image sequences.

Pooled time series representation for mitosis event recognition

This paper proposes a new feature representation for mitotic event detection in time-lapse phase contrast microscopy image sequences of stem cell populations. First, an imaging model-based microscopy image segmentation method is implemented for mitotic candidate extraction. Then, a new feature representation framework based on time series pooling is proposed for sequential events. At last, a support vector machine classifier is utilized for mitotic cell modeling and detection. Different from other feature representations including bag-of-visual-words when using identical underlying feature descriptors, this method can take advantage of temporal relations among frames, the idea is to keep track of how descriptor values are changing over time and summarize them to represent appearance in the cell sequence. The comparison experiments demonstrate the superiority of the proposed method.

Spatiotemporal Joint Mitosis Detection Using CNN-LSTM Network in Time-Lapse Phase Contrast Microscopy Images

We present an approach to jointly detect mitotic events spatially and temporally in time-lapse phase contrast microscopy images. In particular, we combine a convolutional neural network (CNN) and a long short-term memory (LSTM) network to detect mitotic events in patch sequences. The CNN-LSTM network can be trained end-to-end to simultaneously learn convolutional features within each frame and temporal dynamics between frames, without hand-crafted visual or temporal feature design. Owing to the LSTM layer, this approach is able to detect mitotic events in patch sequences of variable length, as well as making use of longer context information among frames in the sequences. To the best of our knowledge, this is the first work to detect mitosis using deep learning in both spatial and temporal domains. Experiments have shown that the CNN-LSTM network can be trained efficiently, and we evaluate this design by applying the network to original raw microscopy image sequences to locate mitotic events both spatially and temporally. The data with which we validate the proposed method include C3H10 mesenchymal and C2C12 myoblastic stem cell populations. Our approach achieved the F score of 98.72% on the C2C12 data set, and the F score of 96.5% on the C3H10 data set. The results on both data sets outperform the traditional graph model-based approaches by a large margin, both in terms of detection accuracy and frame localization accuracy. Furthermore, we have developed a framework to aid humans in annotating mitosis with high efficiency and accuracy in raw phase contrast microscopy images based on the joint detection results using the proposed method. Under this framework, expert level annotations can be obtained in raw phase contrast microscopy image sequences, and the annotations have shown to further improve the training performance of the CNN-LSTM network.

A Semi-Markov Model for Mitosis Segmentation in Time-Lapse Phase Contrast Microscopy Image Sequences of Stem Cell Populations

We propose a semi-Markov model trained in a max-margin learning framework for mitosis event segmentation in large-scale time-lapse phase contrast microscopy image sequences of stem cell populations. Our method consists of three steps. First, we apply a constrained optimization based microscopy image segmentation method that exploits phase contrast optics to extract candidate subsequences in the input image sequence that contains mitosis events. Then, we apply a max-margin hidden conditional random field (MM-HCRF) classifier learned from human-annotated mitotic and nonmitotic sequences to classify each candidate subsequence as a mitosis or not. Finally, a max-margin semi-Markov model (MM-SMM) trained on manually-segmented mitotic sequences is utilized to reinforce the mitosis classification results, and to further segment each mitosis into four predefined temporal stages. The proposed method outperforms the event-detection CRF model recently reported by Huh as well as several other competing methods in very challenging image sequences of multipolar-shaped C3H10T1/2 mesenchymal stem cells. For mitosis detection, an overall precision of 95.8% and a recall of 88.1% were achieved. For mitosis segmentation, the mean and standard deviation for the localization errors of the start and end points of all mitosis stages were well below 1 and 2 frames, respectively. In particular, an overall temporal location error of 0.73 ± 1.29 frames was achieved for locating daughter cell birth events.