Learning Spatio-Temporal Aggregation for Fetal Heart Analysis in Ultrasound Video

Abstract

We investigate recent deep convolutional architectures for automatically describing multiple clinically relevant properties of the fetal heart in Ultrasound (US) videos, with the goal of learning spatio-temporal aggregation of deep representations. We examine multiple temporal encoding models that combine both spatial and temporal features tailored for US video representation. We cast our task into a multi-task learning problem within a hierarchical convolutional model that jointly predicts the visibility, view plane and localization of the fetal heart at the frame level. We study deep convolutional networks developed for video classification, and analyse them for our task by looking at the architectures and the multi-task loss in the specific modality of US videos. We experimentally verify that the developed hierarchical convolutional model that progressively encodes temporal information throughout the network is powerful to retain both spatial details and rich temporal features, which leads to high performance on a real-world clinical dataset.