Optical flow-based filtering for effective presentation of the enhanced vision on a HUD

Abstract

Thermal imaging cameras improve the situational awareness of pilots during the aircraft operation. Nowadays thermal sensors are readily available onboard as the part of the Enhanced Vision System (EVS). While video synthesized using 3D modeling (Synthetic Vision System, SVS) can be easily displayed on a Head-up Display (HUD) due to the presence of the area segmentation data, the projection of the EVS video on a HUD usually results in an image with large bright areas that partially obscure the cockpit view from the cabin crew. This paper is focused on the development of the ClearHUD algorithm for effective presentation of the EVS video on a HUD using the optical flow estimation. The ClearHUD algorithm is based on the optical flow estimation using the video from the SVS and the EVS. The difference of the optical flows is used to detect the obstacles. The areas of the detected obstacles are projected with high intensity, and the remaining regions are filtered using the segmentation from the SVS. The ClearHUD algorithm was implemented in a prototype software for testing using 3D modeling. The optical flow for the SVS is estimated using ray tracing. The optical flow for the EVS is estimated using FlowNet 2.0 convolutional neural network (CNN). The evaluation of the ClearHUD algorithm has proved that it provides a significant increase of brightness of obstacles and reduces the intensity of non-informative areas.