Multi-operator Image Retargeting based on Saliency Object Ranking and Similarity Evaluation Metric

Abstract

Image Retargeting (IR) technology is proposed to flexibly display images on various display devices while protecting the important content of the images undistorted. IR methods mainly use Salient Object Detection (SOD) to obtain important content, however, most existing SOD methods treat multiple salient objects with the same saliency degrees, which makes IR methods assign the same retargeting ratios for different objects and leads to producing information-loss retargeted results. Multi-operator IR demonstrates better generalization than single operator by using multiple operators to find the optimal sequence of operators. Meanwhile, the tremendous processing time limits its practical use. To address these problems, we propose a multi-operator IR method based on Salient Object Ranking (SOR) and Similarity Evaluation Metric (SORSEM-IR), which includes two stages: importance map generation and multi-operator IR. In the first stage, a SOR module with Context-aware Semantic Refinement (SORCSR) is proposed, which extracts the salient instances and infers their saliency ranks with a context-aware semantic refinement module, then the SOR map, face map, and gradient map are fused as the importance map. In the second stage, to speed up multiple operations, a similarity evaluation metric is proposed to measure the similarity between the original image and the seam-removal image by Seam Carving (SC) operation, and switch SC to uniform scaling to meet the aspect ratio when distortion caused by SC arrives at a certain extent. Experimental results show that the SORCSR network achieves state-of-the-art performance on the ASSR dataset subjectively and objectively, and the SORSEM-IR guided by SORCSR can not only protect the salient objects with minimum deformation but also meet human aesthetic perception.