Spatiotemporal Joint Mitosis Detection Using CNN-LSTM Network in Time-Lapse Phase Contrast Microscopy Images

Abstract

We present an approach to jointly detect mitotic events spatially and temporally in time-lapse phase contrast microscopy images. In particular, we combine a convolutional neural network (CNN) and a long short-term memory (LSTM) network to detect mitotic events in patch sequences. The CNN-LSTM network can be trained end-to-end to simultaneously learn convolutional features within each frame and temporal dynamics between frames, without hand-crafted visual or temporal feature design. Owing to the LSTM layer, this approach is able to detect mitotic events in patch sequences of variable length, as well as making use of longer context information among frames in the sequences. To the best of our knowledge, this is the first work to detect mitosis using deep learning in both spatial and temporal domains. Experiments have shown that the CNN-LSTM network can be trained efficiently, and we evaluate this design by applying the network to original raw microscopy image sequences to locate mitotic events both spatially and temporally. The data with which we validate the proposed method include C3H10 mesenchymal and C2C12 myoblastic stem cell populations. Our approach achieved the F score of 98.72% on the C2C12 data set, and the F score of 96.5% on the C3H10 data set. The results on both data sets outperform the traditional graph model-based approaches by a large margin, both in terms of detection accuracy and frame localization accuracy. Furthermore, we have developed a framework to aid humans in annotating mitosis with high efficiency and accuracy in raw phase contrast microscopy images based on the joint detection results using the proposed method. Under this framework, expert level annotations can be obtained in raw phase contrast microscopy image sequences, and the annotations have shown to further improve the training performance of the CNN-LSTM network.