On the learning of image social relevance from heterogeneous social network

Abstract

Social media has been a convenient platform for users to maintain and share messages in different media types (e.g., image and video). Besides the sheer volume of data, various user social behaviors make the data socially related rather than existing independently. The social relevance between images caused by the user activities embeds human cognition towards the images, which is essential in many emerging applications but rarely represented by standard visual features. However, due to the heterogeneous social entities, contents and relationships, social image analysis is a challenging task. In this paper, we propose to decompose the heterogeneous social network into several homogeneous networks, on which the global relevance between social entities can be learned efficiently using a random walk style approach. Furthermore, the image social relevance is determined by propagating the learned relevance to image domain through the links between images and other networks. The proposed image social relevance is seamlessly integrated into several real-word applications, such as image tagging and re-ranking. Our experiment results demonstrate that the image social relevance, which can capture users' cognition on images, is a complementary to visual features, leading to significant performance improvements in both personalized and general applications.