An efficient system using implicit feedback and lifelong learning approach to improve recommendation

Abstract

This paper presents a new technique for contextual item-to-item Collaborative Filtering-based Recommender System, an improved version popularised by e-commerce giant Amazon two decades back. The concept is based on items also-viewed under the same browsing session. Users’ browsing patterns, locations, and timestamps are considered as the context and latent factors for each user. The algorithm computes recommendations based on users’ implicit endorsements by clicks. The algorithm does not enforce the user to log in to provide recommendations and is capable of providing accurate recommendations for non-logged-in users and with a setting where the system is unaware of users’ preferences and profile data (non-logged-in users). This research takes the cue from human lifelong incremental learning experience applied to machine learning on a large volume of the data pool. First, all historical data is gathered from collectable sources in a distributed manner through big data tools. Then, a long-running batch job creates the initial model and saves it to Hadoop Distributed File System (HDFS). An ever-running streaming job loads the model from HDFS and builds on top of it in an incremental fashion. At the architectural level, this resembles the big data mix processing Lambda Architecture. The recommendation is computed based on a proposed equation for a weighted sum between near real-time and historical batch data. Real-time and batch processing engines act as autonomous Multi-agent systems in collaboration. We propose an ensemble method for batch-stream the recommendation engine. We introduce a novel Lifelong Learning Model for recommendation through Multi-agent Lambda Architecture. The recommender system incrementally updates its model on streaming datasets to improve over time.