Edge device-based real-time implementation of CycleGAN for the colorization of infrared video

Abstract

Edge computing has gained significant attention as a pivotal technology for practical implementation of machine learning. Nevertheless, resource-constrained edge devices face challenges in meeting the performance requirements of latency-sensitive applications. Moreover, computationally intensive applications such as CycleGAN pose further hindrances to the practical utilization of machine learning. To overcome these challenges, we proposed a hybrid method that combines the implementation of a lightweight model with optimization in deployment. Initially, a lightweight sub-network is derived from a student network obtained through distillation while considering the trade-off between performance and computational complexity. Subsequently, a quantization model is deployed for low-precision inference, resulting in a substantial reduction of inference time. Finally, we apply this approach to a real-life application: a real-time headset for color night vision from infrared videos. For this purpose, three datasets of image pairs comprising long-wave infrared, visible light RGB, and up to near-infrared images are collected. Using these datasets, a lightweight implementation of the CycleGAN model is trained to translate infrared images to RGB images. To ensure efficiency, the model is deployed in a low-precision inference manner using C＋＋ on three different hardware platforms: Jetson Xavier NX, RK3399 Pro, and Raspberry Pi, each with distinct hardware architectures. Image quality and model efficiency are thoroughly analyzed. Experimental results demonstrate that our hybrid method drastically reduces inference time from 112 ms to 17 ms per frame at a resolution of 256 × 256 on the Jetson Xavier NX platform. This improvement is accompanied by only a slight degradation in image quality, enabling a real-time video frame rate close to 60 fps, thereby meeting the requirements for a real-world real-time headset for color night vision applications.