Image Sentiment Analysis Research Articles

Enhancing image sentiment analysis: A user-centered approach through user emotions and visual features

To bridge the gap between user understanding of emotions and image representation, this paper amalgamated a variety of user features to formulate an image sentiment analysis model. Through controlled user experiments, participants were engaged in discerning sentiment from abstract painting images. User visual features were derived from eye-tracking data collected during the experiment. This paper employed the FFM (Five Factor Model) and POMS (Profile of Mood States) methodologies to gather personality and mood data, which were utilized as user emotional features. Additionally, fundamental components of abstract painting were captured as basic image features. By synergizing features across these three dimensions, the study introduced the "User-Feature-Enhanced-Image-Sentiment-Model" (UFE-ISM) for image sentiment recognition. The research findings underscore the model's superior performance in sentiment classification on abstract painting. Notably, both basic image features and user visual features exhibited significant contributions to the image sentiment classification model.

Object aroused emotion analysis network for image sentiment analysis

From cognitive psychology, objects are closely related to emotions, and inherently possess the ability to arouse human emotions. Hence, fully utilizing the relationships between objects and emotions can help achieve more accurate visual emotion recognition. In this paper, we propose a novel object aroused emotion analysis network to realize image sentiment classification by investigating the interactions between objects and emotions. To quantify the various emotion potencies of each object, a novel object emotion distribution module is proposed to explore the mapping among objects and emotions, and quantitatively demonstrate how various objects arouse different emotions. An object emotion-modeling mapping module is proposed to analyze the effect of objects and object combinations with emotion information on image sentiment; this module maps common objects to emotion dimensions and improves image sentiment classification with abundant object combination information. Then, we fuse the object-emotion mapping relation and multi-model features using BiGRU, thus realizing more accurate emotion recognition. Extensive experiments on widely used emotion datasets prove that our proposed method achieves excellent performance and outperforms most state-of-the-art image sentiment classification methods.

ClKI: closed-loop and knowledge iterative via self-distillation for image sentiment analysis

ClKI: closed-loop and knowledge iterative via self-distillation for image sentiment analysis

Color Enhanced Cross Correlation Net for Image Sentiment Analysis

Automatic analysis of image sentiment has gained considerable attention with the increasing throughput of user-generated visual contents online. Recently, researchers generally tend to design different Convolutional Neural Networks (CNNs) to extract image content features for sentiment analysis. However, they underestimated the importance of image color, which has been proved very crucial for image sentiment expressing by psychology and art theory. Moreover, we further observe that the coordination of content and color is the main form of image sentiment expressing. Different combinations of content and color could express extremely different sentiments. To that end, in this paper, we propose a Color Enhanced Cross Correlation Net (CECCN), a novel architecture for image sentiment analysis that not only leverages contents and colors simultaneously, but also takes their correlations into consideration. Specifically, we first use a pre-trained CNN to extract content features and color moment to collect color features from multiple color spaces. Then, we propose a novel Cross Correlation (CC) method to model the correlations between content features and color features with attention mechanism and sequence convolution, in which sentiment expressing of content and color can be enhanced by each other. Finally, we integrate these two types of information for better image sentiment analysis. Extensive experiments on two popular and well-studied benchmark datasets demonstrate the superiority and rationality of our proposed CECCN.

MPCM: Multi-modal User Portrait Classification Model Based on Collaborative Learning

A social-media user portrait is an important means of improving the quality of an Internet information service. Current user profiling methods do not discriminate the emotional differences of users of different genders and ages on social media against a background of multi-modality and a lack of domain sentiment labels. This paper adopts the sentiment analysis of images and text to improve label classification, incorporating gender and age differences in the sentiment analysis of multi-modal social-media user profiles. In the absence of domain sentiment labels, instance transfer learning technology is used to express the learning method with the sentiment of text and images; the semantic association learning of multi-modal data of graphics and text is realized; and a multi-modal attention mechanism is introduced to establish the hidden image and text. Alignment relationships are used to address the semantic and modal gaps between modalities. A multi-modal user portrait label classification model (MPCM) is constructed. In an analysis of the sentiment data of User users on Facebook, Twitter, and News, the MPCM method is compared with the naive Bayes, Latent Dirichlet Allocation, Tweet-LDA and LUBD-CM(3) methods in terms of accuracy, precision, recall and the FL-score. At a 95% confidence, the performance is improved by 1% to 4% by using the MPCM method.

Art appreciation model design based on improved PageRank and ECA-ResNeXt50 algorithm.

Image sentiment analysis technology can predict, measure and understand the emotional experience of human beings through images. Aiming at the problem of extracting emotional characteristics in art appreciation, this article puts forward an innovative method. Firstly, the PageRank algorithm is enhanced using tweet content similarity and time factors; secondly, the SE-ResNet network design is used to integrate Efficient Channel Attention (ECA) with the residual network structure, and ResNeXt50 is optimized to enhance the extraction of image sentiment features. Finally, the weight coefficients of overall emotions are dynamically adjusted to select a specific emotion incorporation strategy, resulting in effective bimodal fusion. The proposed model demonstrates exceptional performance in predicting sentiment labels, with maximum classification accuracy reaching 88.20%. The accuracy improvement of 21.34% compared to the traditional deep convolutional neural networks (DCNN) model attests to the effectiveness of this study. This research enriches images and texts' emotion feature extraction capabilities and improves the accuracy of emotion fusion classification.

EERCA-ViT: Enhanced Effective Region and Context-Aware Vision Transformers for image sentiment analysis

Different parts of an image have a strong or weak guiding effect on emotions. The key to emotion recognition of images is to fully exploit the regions associated with emotions. Therefore, this paper proposes a visual sentiment classification model with two branches based on visual transformer, termed as Enhanced Effective Region and Context-Aware Vision Transformers (EERCA-ViT). This model includes a primary branch and an auxiliary branch. The primary branch simulates interdependencies between patches by squeezing and stimulating patches (P-SE), thereby highlighting effective region features in the global feature. The auxiliary branch removes feature patches that have been tagged by the primary branch through the context-aware module (CAM), forcing the network to discover different sentiment discriminative regions. At the same time, a two-part loss function is constructed to improve the robustness of the model. Finally, extensive experiments on six benchmark datasets show that the proposed method outperforms the state-of-the-art image sentiment analysis methods. Furthermore, the effectiveness of the different modules of the framework (P-SE and CAM) has been well demonstrated through extensive comparative experiments.

Sentiment analysis on images using convolutional neural networks based Inception-V3 transfer learning approach

Sentiment analysis has become a precious tool for businesses because it can be used in so many ways: to find out what customers think about products and services, to build customer relationships and loyalty, to improve customer service, and to use emotional marketing. Over the last several years, developing an end-to-end image sentiment analysis approach has significantly emphasized transfer learning methods. Deep learning algorithms have been proven to achieve remarkable outcomes across a broad spectrum of applications. Examining feelings conveyed by images is complex, but there is much space for development. A technique known as Inception-v3 that can readily focus on large portions of the body, such as a person's face, offers a significant advantage compared to the work that was done in the past. This study makes use of Inception-v3, which is a well-known deep convolutional neural network, in addition to extra deep characteristics, to increase the performance of image categorization. A CNN-based Inception-v3 architecture is employed for emotion detection and classification. The datasets CK+, FER2013, and JAFFE are used in this process. The findings are also compared with various well-known machine learning approaches, and the results obtained by the suggested model are superior. The research indicates that the proposed method can reach an accuracy level of 99.5%. The proposed approach can be used in many business applications such as Information Management, Sales, Marketing, User Interaction, Healthcare, Education, Finance, Public Monitoring, Digital PR, etc.

Context Aware Image Sentiment Classification using Deep Learning Techniques

Objectives: To propose context aware sentiment classification using deep learning techniques. Methods: We used EfficientNetB-7 deep learning framework for caption generation for the input image and to classify the sentiment of generated caption using machine learning techniques. First, we employ several real-time and synthetic image datasets, then apply pre-processing and normalization for data balancing. Then efficient module implementation for feature extraction and selection using convolutional and pooling layers were done. Despite this proceeding, it generates the caption for respective images. The various feature extraction and selection Natural Language Processing (NLP) techniques such as TF-IDF, lemmas, dependency and correlational features have been used and classify the sentiment label using attention model and greedy approach. Finally, generating the blue score for the entire testing dataset and show the effectiveness of the proposed system. Findings: Our model gives higher accuracy with different deep learning techniques which is demonstrated in result section. The proposed model archives 73.80% average accuracy for EMOTIC dataset. The module has evaluated with different features and deep learning classification algorithms proposed earlier. Novelty: This research is the collaboration of Deep learning and machine learning classification techniques. We first extract the visual features from the input image using deep learning and classify with machine learning with the collaboration of NLP processes. We also carried out various feature extraction techniques such as Ngram, dependency features, co-relational features and determined the sentiment of generated captions. Keywords: Image Sentiment analysis; Emotic dataset; CNN; EfficientNetB7; Attention based LSTM; GRU

Polarity-aware attention network for image sentiment analysis

Polarity-aware attention network for image sentiment analysis

Sentiment Analysis of Image with Text Caption using Deep Learning Techniques.

People are actively expressing their views and opinions via the use of visual pictures and text captions on social media platforms, rather than just publishing them in plain text as a consequence of technical improvements in this field. With the advent of visual media such as images, videos, and GIFs, research on the subject of sentiment analysis has expanded to encompass the study of social interaction and opinion prediction via the use of visuals. Researchers have focused their efforts on understanding social interaction and opinion prediction via the use of images, such as photographs, films, and animated GIFs (graphics interchange formats). The results of various individual studies have resulted in important advancements being achieved in the disciplines of text sentiment analysis and image sentiment analysis. It is recommended that future studies investigate the combination of picture sentiment analysis and text captions in more depth, and further research is necessary for this field. An intermodal analysis technique known as deep learning-based intermodal (DLBI) analysis is discussed in this suggested study, which may be used to show the link between words and pictures in a variety of scenarios. It is feasible to gather opinion information in numerical vector form by using the VGG network. Afterward, the information is transformed into a mapping procedure. It is necessary to predict future views based on the information vectors that have been obtained thus far, and this is accomplished through the use of active deep learning. A series of simulation tests are being conducted to put the proposed mode of operation to the test. When we look at the findings of this research, it is possible to infer that the model outperforms and delivers a better solution with more accuracy and precision, as well as reduced latency and an error rate, when compared to the alternative model (the choice).

Ensemble-based face expression recognition approach for image sentiment analysis

<span>Sentiment analysis based on images is an evolving area of study. Developing a reliable facial expression recognition (FER) device remains a difficult challenge as recognizing emotional feelings reflected in an image is dependent on a diverse set of factors. This paper presented an ensemble-based model for FER that incorporates multiple classification models: i) customized convolutional neural network (CNN), ii) ResNet50, and iii) InceptionV3. The model averaging ensemble classifier method is used to ensemble the predictions from the three models. Subsequently, the proposed FER model is trained and tested on a dataset with an uncontrolled environment (FER-2013 dataset). The experiment demonstrated that ensembling multiple classifiers outperformed all single classifiers in classifying positive and neutral expressions (91.7%, 81.7% and 76.5% accuracy rate for happy, surprise, and neutral, respectively). However, when classifying disgust, anger, and sadness, the ResNet50 model alone is the better choice. Although the Custom CNN performs the best in classifying fear expression (55.7% accuracy), the proposed FER model can still classify fear expression with comparable performance (52.8% accuracy). This paper demonstrated the potential of using the ensemble-based method to enhance the performance of FER. As a result, the proposed FER model has shown a 72.3% accuracy rate.</span>

Chain centre loss: A psychology inspired loss function for image sentiment analysis

In this paper, we present a novel loss function called a chain center loss for image sentiment analysis. The proposed loss is derived from two famous deep metric loss functions, a center loss and a triplet loss, for latent space construction. Specifically, inspired by both previous loss functions, the proposed loss supervises the local and global spatial distributions. As for the local spatial distribution, the chain center loss generates center points by the Gaussian distribution and integrates visual features extracted from the images of the same sentiment. Moreover, different from the previous deep metric measures, we newly introduce sentiment correlations revealed by psychology theories into the chain center loss to capture the global spatial distribution. Concretely, the proposed loss applies anchor-related-negative triplets to the loss computation inspired by the triplet loss and updates the center points with consideration of the correlations among sentiments. As a result, with the use of the proposed loss, a latent space that strongly represents sentiment correlations can be constructed with high robustness. Improvement in the performance of the chain center loss was confirmed from experimental results.

IMAGE SENTIMENTAL ANALYSIS: AN OVERVIEW

Visual content, such as photographs and video, contains not only objects, locations, and events, but also emotional and sentimental clues. On social networking sites, images are the simplest way for people to communicate their emotions. Images and videos are increasingly being used by social media users to express their ideas and share their experiences. Sentiment analysis of such large-scale visual content can aid in better extracting user sentiments toward events or themes, such as those in image tweets, so that sentiment prediction from visual content can be used in conjunction with sentiment analysis of written content. Despite the fact that this topic is relatively new, a wide range of strategies for various data sources and challenges have been created, resulting in a substantial body of study. This paper introduces the area of Image Sentiment Analysis and examines the issues that it raises. A description of new obstacles is also included, as well as an assessment of progress toward more sophisticated systems and related practical applications, as well as a summary of the studys findings.

Image Sentiment Analysis via Active Sample Refinement and Cluster Correlation Mining

Training an effective image sentiment analysis model using high-quality samples and the implicit cross-modal semantics among heterogeneous features is still challenging. To address this problem, we propose an active sample refinement (ASR) strategy to obtain sufficient high-quality images with definite sentiment semantics. We mine the cluster correlation among the heterogeneous SENet features. Discriminative cross-modal semantics is generated to train an effective but robust image classifier. Ensemble learning is employed to further boost performance. Our method outperforms other competitive baselines, demonstrating its effectiveness and robustness. Meanwhile, the ASR strategy is a useful supplement to the current data augmentation method.

Research on image sentiment analysis technology based on sparse representation

Many methods based on deep learning have achieved amazing results in image sentiment analysis. However, these existing methods usually pursue high accuracy, ignoring the effect on model training efficiency. Considering that when faced with large-scale sentiment analysis tasks, the high accuracy rate often requires long experimental time. In view of the weakness, a method that can greatly improve experimental efficiency with only small fluctuations in model accuracy is proposed, and singular value decomposition (SVD) is used to find the sparse feature of the image, which are sparse vectors with strong discriminativeness and effectively reduce redundant information; The authors propose the Fast Dictionary Learning algorithm (FDL), which can combine neural network with sparse representation. This method is based on K-Singular Value Decomposition, and through iteration, it can effectively reduce the calculation time and greatly improve the training efficiency in the case of small fluctuation of accuracy. Moreover, the effectiveness of the proposed method is evaluated on the FER2013 dataset. By adding singular value decomposition, the accuracy of the test suite increased by 0.53%, and the total experiment time was shortened by 8.2%; Fast Dictionary Learning shortened the total experiment time by 36.3%.

Sentiment Analysis from Images using VGG19 based Transfer Learning Approach

Using image sentiment analysis rather than text sentiment analysis to make different judgments has grown increasingly popular recently. In recent years, transfer learning techniques have been employed extensively in developing end-to-end image sentiment analysis methodologies, which is expected to continue. Deep learning algorithms have produced outstanding outcomes in a variety of applications. Even though image-based sentiment analysis is tricky, there seems to be much space for progress. Research presented here suggests that a VGG-19-based approach, which can easily be used to concentrate on vast body areas, such as the face, offers a significant gain over earlier studies. This research aims to improve image categorization performance by using the well-known deep convolutional neural network, VGG19, and other deep features. Emotion detection and classification with diverse emotions utilizing a VGG-19 (CNN-based) architecture can be easily performed using CK+, FER2013, and JAFFE datasets. The results show that the proposed method can reach accuracy up to 99%. It can study a person's emotional habits and psychological condition using the approach.

Gabor CNN Based Intelligent System for Visual Sentiment Analysis of Social Media Data on Cloud Environment

Social media contains a plethora of information in the form of text, images, videos, and other data. Users across the globe are increasingly sharing their data on various social media platforms. Sentiment analysis of data, such as text, images, and videos are widely used to understand the feelings of users. In recent years, the convolutional neural network (CNN) has been extensively applied for various applications. The cloud computing environment is a popular service due to its reliability, availability, and easy software integration. However, CNN models are deep neural networks that have a high computational cost. There is a need for CNN models which utilize lesser computational resources especially when these models are deployed in a cloud environment due to the remote physicality of servers, resource optimization, and infrastructure cost reduction. In this research, Gabor filters are integrated with CNN models to improve image sentiment analysis in a cloud environment, with advantages such as the reduction in computation energy and time, the elimination of the need for pre-trained models, and a perceived accuracy improvement. Two variants of Gabor-CNN (G-CNN) models with a different number of pooling and normalization layers are developed. The proposed G-CNN is trained and tested using <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">five</i> standard databases as SentiBank, Twitter, MVSO, MultiView_I, and MultiView_II. Maximum classification accuracies of 91.71%, 92.52%, 97.39%, 90.88%, and 91.31% are obtained on SentiBank, Twitter, MVSO, MultiView_I, and MultiView_II databases respectively using the developed models. The proposed G-CNN model has provided an accuracy of 92.76% on average.

Intermodal Sentiment Analysis for Images with Text Captions Using the VGGNET Technique

More individuals actively express their opinions and attitudes in social media through advanced improvements such as visual content and text captions. Sentiment analysis for visuals such as images, video, and GIFs has become an emerging research trend in understanding social involvement and opinion prediction. Numerous individual researchers have obtained good progress in outcomes for text sentiment analysis and image sentiment analysis. The combination of image sentiment analysis with text caption analysis needs more research. This article presents a VGG Network-based Intermodal Sentiment Analysis Model (VGGNET-ISAM) for transferring the connection between texts to images. A mapping process is developed using the VGG Network for gathering the opinion information as numerical vectors. The Active Deep Learning (ADL) classifier is used for opinion prediction from the obtained information vectors. Simulation experiments are carried out to evaluate the proposed approach. The findings show that the model outperforms and gives better solutions with high accuracy, precision with low delay, and low error rate.

Social Image Sentiment Analysis by Exploiting Multimodal Content and Heterogeneous Relations

In the circumstance of social big data, sentiment analysis is attracting increasing attention for its capacity in understanding individuals' attitudes and feelings. Traditional sentiment analysis methods focus on single modality and become ineffective as enormous data are emerging on the social websites with multiple manifestations. In this article, multimodal learning approaches are proposed to capture the relations between image and text, which only stay at the region level and ignore the fact that the channels are also closely correlated with the semantic information. In addition, social images in the social platforms are closely connected by various types of relations, which are also conducice to sentiment classification but neglected by most existing works. In this article, we propose an attention-based heterogeneous relational model to improve the multimodal sentiment analysis performance by incorporating rich social information. Specifically, we propose a progressive dual attention module to capture the correlations between image and text, and then learn the joint image-text representation from the perspective of content information. A channel attention schema is proposed here to highlight semantically rich image channels and a region attention schema is further designed to highlight the emotional regions based on the attended channels. After that, we construct a heterogeneous relation network and extend graph convolutional network to aggregate the content information from social contexts as complements to learn high-quality representations of social images. Our proposal is thoroughly evaluated on two benchmark datasets, and experimental results demonstrate the superiority of the proposed model.