Two-stage cross-fusion network for stereo event-based depth estimation

Abstract

Drawing inspiration from biological retina, event camera uses dynamic vision sensor to asynchronously capture pixel intensity changes. Event camera provides a series of advantages, including high dynamic range, high temporal resolution, lack of motion blur, and low power consumption, which make it suitable candidate for depth estimation, specially in challenging scenarios with fast motion and low-lighting conditions. However, due to the sparse nature of event data, it is troublesome to estimate dense disparity map from them. Although it is plausible to estimate disparity map from edges of a structure from where most events are captured, estimating dense disparity is challenging in the areas where events rarely occur. In this study, we develop a stereo matching network for dense disparity estimation, which effectively fuses reconstructed image features with event to estimate disparity. To this end, we propose a two-stage cross-fusion network (TCFNet) to estimate disparity by fusing event and image features with two cascaded fusion mechanisms. First, we propose cross-SPADE fusion, based on the spatially-adaptive denormalization mechanism (SPADE), which modulates event features with image features, and image features with event features. In the second stage, the modulated features are further fused using the cross-gating mechanism, which uses the multi-axis gated MLP (gMLP). Experiments on both real and synthetic datasets indicate that the proposed two-stage cross-fusion network outperforms the state-of-the-art event-based stereo matching networks empirically and visually.