Transfer-based adaptive tree for multimodal sentiment analysis based on user latent aspects

Abstract

Multimodal sentiment analysis benefits various applications, such as human–computer interaction and recommendation systems. It infers the user-based sentiments using modals, enriched by neutrality and ambivalence empirical. Although researchers affirm the association between cognitive cues and emotional manifestations, most current multimodal sentiment analytical approaches disregard user-specific aspects. To tackle this issue, we devise a novel method to perform multimodal sentiment prediction using cognitive cues. Our cognitive-aware framework constructs an adaptive tree by hierarchically dividing users and then trains the Long short-term memory submodels, utilizing an attention-based fusion to transfer cognitive-oriented knowledge within the tree. Subsequently, the framework consumes the conclusive agglomerative knowledge from the adaptive tree to predict final sentiments. We also develop a dynamic dropout method to regularize data of pertinent groups during hierarchical training. The empirical results on real-world datasets determine that our proposed model for sentiment prediction can surpass trending baselines. Moreover, compared to other ensemble approaches, the proposed transfer-based algorithm can better utilize the latent cues and foster the prediction outcomes, benefitting from multiple categorization strategies. Based on the given extrinsic and intrinsic analysis results, we note that compared to other theoretical-based techniques, the proposed hierarchical categorization model can better group the users within the adaptive tree.