Deep learning for software-based turbulence mitigation in long-range imaging

Abstract

Optical imaging over long horizontal paths often suffers from the effects of atmospheric turbulence. Dynamic density variations in the air result in random spatiotemporally varying shifts and blurs in the recorded images. Software based turbulence mitigation algorithms provide a means of computationally reducing these turbulence effects in video. They provide camera operators with sharper and more stable imagery which supports them on visual recognition and identification tasks. Most turbulence mitigation algorithms rely on a form of temporal low pass filtering to remove the turbulence induced fluctuations. This filtering also suppresses high spatial frequency information and thus limits the ability of these algorithms to recover fine details. Here we propose a turbulence mitigation algorithm which employs a deep neural network to recover high spatial frequency information from turbulence degraded video. The proposed algorithm builds on our previous approach, where frame-to-frame estimates of image shifts are used to combine multiple frames. This approach is amended by using a deep neural network to deblur output images. For the related task of single image super-resolution, we have previously shown that such neural networks provide state-of-the-art image reconstruction performance. Here our neural network is trained using semi-synthetic image sequences of static scenes with simulated turbulence. We show that our deep learning based approach provides on semi-synthetic test data a substantial performance increase compared to our previous sharpening approach. We also apply our algorithm to real long range imagery of ships at sea and find a perceptually similar improvement in image quality as for semi-synthetic data.