MEMF: Multi-entity multimodal fusion framework for sales prediction in live streaming commerce

Abstract

Live streaming commerce thrives with a rich tapestry of multimodal information that intertwines with various entities, including the anchor, the commodities, and the live streaming environment. Despite the wealth of data at hand, the synthesis and analysis of this information to predict sales remains a significant challenge. This study introduces a framework for multi-entity multimodal fusion, which is characterized by the effective synthesis of multimodal data and its prioritization of entity-level fusion, thereby providing a comprehensive feature representation for improving predictive performance. In addressing the multimodal data associated with a diverse range of products, our framework improves the Transformer architecture to initially capture the intra-product modal features and subsequently integrate the inter-product features. Data experiments are conducted on a real-world dataset from Taobao Live. The framework outperforms both traditional machine learning methods and state-of-the-art multimodal fusion methods, which affirms its value as a robust decision-support tool for sales prediction, enabling more accurate pre-event predictions and strategic planning. We also examine the impact of different types of information in accurate sales prediction. It is found that harnessing a comprehensive suite of data leads to optimal performance across all evaluation metrics. Commodity-related data is primary factor in determining the prediction accuracy, followed by video data and streaming room-related data, providing insights regarding the resource allocation for collecting and analyzing multimodal data from live streaming platforms.