Hybrid image analysis model for hashtag recommendation through the use of deep learning methods

Abstract

Content tags of a given image are the basic mechanism used in social networking services. One of the most commonly used websites like Instagram, Facebook, or Twitter is the hashtag. In this paper, we propose a hybrid solution for automatic image description extraction. A network architecture consists of a module for removing main objects from the image for additional classification of other elements on the background via U-NET (neural model designed for segmentation). Obtained images are classified by multi-branch convolutional networks. The results of such a classification are used to describe the probability of belonging to particular classes. Then, the probability of belonging to each class is used to find related linguistic values through the Skip-Gram module. The main advantage of the solution is the possibility of extending the network with additional branches to increase the number of main classes of objects. The proposal was analyzed through the object’s classification, the use of learning transfer, and image description on two datasets, i.e. Animals-10 and HARRISON16. Experimental results show that the proposal is a very flexible solution for extending the model to other classes without the need to overtrain the entire architecture and reach higher classification accuracy than other models for social networking services.