Hierarchical Convolutional Neural Network for Infrared Image Super-Resolution

Abstract

Due to the high price of thermal imaging sensors, methods for high quality upscaling of infrared images, acquired from low-resolution inexpensive IR-cameras become in high demand. One of the very promising branches of such kinds of methods is base on super-resolution (SR) techniques that exploit convolutional neural networks (CNN), which are developed rapidly for the last decade. During the review of existing solutions, we found that most of the super-resolution neural networks are intended for the upscaling of images in the visible spectrum band. Among them, the BCLSR network has proven to be one of the best solutions that ensure a very high quality of image upscaling. Thus, we selected this network for further investigation in the current paper. Namely, in this research, we trained and tested the BCLSR network for upscaling of far-infrared (FIR) images for the first time. Moreover, inspired by the BCLSR architecture, we proposed our own neural network, which defers from the BCLSR by the absence of recursive and recurrent layers that were replaced by series-connected Residual- and parallel-connected Inception-like blocks correspondingly. During the tests, we found that the suggested modifications permit to increase the network inference speed almost twice and even improve the quality of upscaling by 0,063 dB compared to the basic BCLSR implementation. Networks were trained and tested using the CVC-14 dataset that contains FIR images acquired at the night. We used data augmentation with random dividing dataset images onto 100×100 pixel patches and with subsequent application random brightness, contrast, and mirroring to the obtained patches. The training procedure was performed in a single cycle with single increase and decrease of the learning rate and used the same parameters for the proposed and the BCLSR networks. We employed the Adam optimizer for the training of both networks. Nevertheless, the proposed model has more parameters (2,7 М) compared to the BCLSR (0,6 М), both of the networks can be considered as the small ones, and thus can be used in applications for conventional personal computers, as well as in embedded solutions. The direction of the further research can be focused on the improvements of the proposed network architecture by introducing new types of layers as well as on the modifying of hyperparameters of the used layers. The quality of the upscaling can be increased also by using other loss functions and by the change of learning rate-varying strategies.