Automatic Key-frame Selection Research Articles

Good Keyframes to Inpaint.

Diminished Reality (DR) propagates pixels from a keyframe to subsequent frames for real-time inpainting. Keyframe selection has a significant impact on the inpainting quality, but untrained users struggle to identify good keyframes. Automatic selection is not straightforward either, since no previous work has formalized or verified what determines a good keyframe. We propose a novel metric to select good keyframes to inpaint. We examine the heuristics adopted in existing DR inpainting approaches and derive multiple simple criteria measurable from SLAM. To combine these criteria, we empirically analyze their effect on the quality using a novel representative test dataset. Our results demonstrate that the combined metric selects RGBD keyframes leading to high-quality inpainting results more often than a baseline approach in both color and depth domains. Also, we confirmed that our approach has a better ranking ability of distinguishing good and bad keyframes. Compared to random selections, our metric selects keyframes that would lead to higher-quality and more stably converging inpainting results. We present three DR examples, automatic keyframe selection, user navigation, and marker hiding.

Tracking better, tracking longer: automatic keyframe selection in model-based laparoscopic augmented reality.

We present a novel automatic system for markerless real-time augmented reality. Our system uses a dynamic keyframe database, which is required to track previously unseen or appearance-changing anatomical structures. Our main objective is to track the organ more accurately and over a longer time frame through the surgery. Our system works with an offline stage which constructs the initial keyframe database and an online stage which dynamically updates the database with new keyframes automatically selected from the video stream. We propose five keyframe selection criteria ensuring tracking stability and a database management scheme ensuring real-time performance. Experimental results show that our automatic keyframe selection system based on a dynamic keyframe database outperforms the baseline system with a static keyframe database. An increase in number of tracked frames without requiring surgeon input is observed with an average improvement margin over the baseline of 11.9%. The frame rate is kept at the same values as the baseline, close to 50 FPS, and rendering remains smooth. Our software-based tracking system copes with new viewpoints and appearance changes during surgery. It improves surgical organ tracking performance. Its criterion-based architecture allows a high degree of flexibility in the implementation, hence compatibility with various use cases.

An Automatic Key-Frame Selection Method for Monocular Visual Odometry of Ground Vehicle

In response to the rapid change of attitude in complex situations such as sharp turns and camera jitter, visual odometry/visual simultaneous localization and mapping (VO/VSLAM) often requires a high camera frame rate. However, the high frame rate of the camera poses a great challenge to real-time VO/VSLAM, especially when the vehicle travels along a relatively flat trajectory which requires higher real-time performance. Therefore, in this paper, we propose an automatic method for key-frame selection based on the motion state of the vehicle, which can reduce data redundancy and improve the real-time performance and robustness of VO/VSLAM. First, a pyramid-layered Kanade-Lucas-Tomasi (KLT) algorithm is used to track the feature points, and the five-point method and RANSAC are used to calculate the essential matrix. Second, two-step decomposition is used to calculate the change of inter-frame attitude, and then, key-frames are automatically selected based on the motion state of the vehicle predicted by attitude change within a short time interval. To evaluate the method, we conduct extensive experiments on real data and Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset based on monocular visual odometry. Then, we made qualitative and quantitative evaluations from the comparison with the reference trajectory, relative error (RE), root mean square error (RMSE), and absolute trajectory error (ATE). The results indicate that our method can improve the real-time performance while ensuring accuracy, with the data redundancy reduced by about 40%-60%. In addition, the performance of our method is further verified by comparison with the current representative ORB-SLAM.

Keyframe Selection Framework Based on Visual and Excitement Features for Lifelog Image Sequences

Keyframe selection is the process of finding a set of representative frames from an image sequence. We aim to achieve an automatic keyframe selection. The main problem is that the accuracy of keyframe selection is highly subjective to each particular user. To deal with this problem, we propose a VIEMO keyframe selection framework based on visual and excitement features that consists of two integrated modules, namely; event segmentation and keyframe selection. Firstly, scene change detection algorithm was applied for an event segmentation. Later, visual features which are contrast, color variance, sharpness, noise and saliency along with excitement features from a biosensor are used to filter keyframe that closely matches with user selection keyframe. Two different fusion scheme which are flat and hierarchical fusion were also investigated. To evaluate the quality of keyframe from the proposed method, we present an evaluation techniques which grades the quality of the keyframe automatically. Even when the keyframe does not exactly match with the keyframe selected by the user, the degree of acceptance calculated from visual similarity is provided. Experimental results showed that keyframe selection using only visual features yielded an acceptance rate of \(74.16 \,\%\). Our proposed method achieves a higher acceptance rate of \(83.71\,\%\). Moreover, the acceptance rate was improved by the average of \(9.55\,\%\) in all participants. Therefore, our framework provides a potential solution to this subjective issue for keyframe selection in lifelog image sequences selection.

Video super-resolution based on automatic key-frame selection and feature-guided variational optical flow

This paper proposes a new video super-resolution method based on feature-guided variational optical flow. The key-frames are automatically selected and super-resolved using a method based on sparse regression. To overcome the blocking artifacts and deal with the case of small structures with large displacement, an efficient method based on feature-guided variational optical flow is used to super-resolve the non-key-frames. Experimental results show that the proposed method outperforms the existing benchmark in terms of both subjective visual quality and objective peak signal-to-noise ratio (PSNR). The average PSNR improvement from the bi-cubic interpolation is 7.15dB for four datasets. Thanks to the flexibility of designed automatic key-frame selection and the validness of feature-guided variational optical flow, the proposed method is applicable to various practical video sequences.