Abstract
Multi-task learning has been established as an important machine learning framework for leveraging shared knowledge among multiple different but related tasks, with the generalization performance of models enhanced. As a promising learning paradigm, multi-task learning has been widely adopted by various real-world applications, such as recommendation systems. Multi-gate Mixture-of-Experts (MMoE), a well-received multi-task learning method in industry, based on the classic and inspiring Mixture-of-Experts (MoE) structure, explicitly models task relationships and learns task-specific functionalities, generating significant improvements. However, in our applications, negative transfer, which confuses considerable existing multi-task learning methods, is still observed to happen to MMoE. In this paper, an in-depth empirical investigation into negative transfer is launched. And it reveals that, incompetent experts, which play fundamental roles under the learning framework of MoE, are the key technique bottleneck. To tackle this dilemma, we propose the Calibrated Mixture of Insightful Experts (CMoIE), with three novel modules (Conflict Resolution, Expert Communication, and Mixture Calibration), customed for multi-task learning. Hence a group of insightful experts are constructed with enhanced diversity, communication and specialization. To validate the proposed method CMoIE, experiments are conducted on three public datasets and one real-world click-through-rate prediction dataset we construct based on traffic logs collected from a large-scale online product recommendation system. Our approach yields best performance across all of these benchmarks, demonstrating the superiority of it.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.