Horizontal-to-Vertical Video Conversion

Abstract

At this blooming age of social media and mobile platform, mass consumers are migrating from horizontal video to vertical contents delivered on hand-held devices. Accordingly, revitalizing the exposure of horizontal video becomes vital and urgent, which is hereby tackled by our automated horizontal-to-vertical (abbreviated as H2V) video conversion framework. Essentially, the {\it \textbf{H2V}} framework performs subject-preserving video cropping instantiated in the proposed Rank-SS module. Rank-SS incorporates object detection to discover candidate subjects, from which we select the primary subject-to-preserve leveraging location, appearance, and salient cues in a convolutional neural network. In addition to converting horizontal videos vertically by cropping around the selected subject, automatic shot detection and multi-object tracking are integrated into the {\it \textbf{H2V}} framework to accommodate long and complex videos. To develop {\it \textbf{H2V}} systems, we collect an {\it \textbf{H2V-142K}} dataset containing 125 videos (132K frames) and 9,500 cover images annotated with primary subject bounding boxes. On {\it \textbf{H2V-142K}} and public object detection datasets, our method demonstrates promising results on the subject selection comparing to the related solutions. Furthermore, our {\it \textbf{H2V}} framework is industrially deployed hosting millions of daily active users and exhibits favorable H2V conversion performance. By making this dataset as well as our approach publicly available, we wish to pave the way for more horizontal-to-vertical video conversion research. Our collected H2V-142K dataset is available atH2V-142K website.