MHaDiG: A Multilingual Humor-aided Multiparty Dialogue Generation in multimodal conversational setting

Abstract

Building a multimodal humor-aided dialogue generation system in a multilingual setting is a very challenging task and it could even be more challenging in the Hindi language because of its morphological richness and free word order. Existing works only solved the problem of unimodal humorous dialogue generation, not humor-aided, in a monolingual setting. However, due to the tremendous growth in multimodal and multilingual content, there is a great demand to build multimodal humor-aided systems that support multilingual information access. To this end, in this paper, we propose a deep learning-based multimodal sentiment and emotion-aware humor-aided multiparty dialogue generation in multilingual (Hindi and English) setting. We evaluate our proposed approach (MHaDiG) on the recently released Sentiment, Humor, and Emotion-aware Multilingual Multimodal Multiparty Dataset (SHEMuD). Experimental results show the efficacy of MHaDiG and show the effect of multilingualism over monolingualism in the multimodal conversational setting.