Evaluation Of Recommender Systems Research Articles

Our Model Achieves Excellent Performance on MovieLens: What Does It Mean?

A typical benchmark dataset for recommender system (RecSys) evaluation consists of user-item interactions generated on a platform within a time period. The interaction generation mechanism partially explains why a user interacts with (e.g., like, purchase, rate) an item, and the context of when a particular interaction happened. In this study, we conduct a meticulous analysis of the MovieLens dataset and explain the potential impact of using the dataset for evaluating recommendation algorithms. We make a few main findings from our analysis. First, there are significant differences in user interactions at the different stages when a user interacts with the MovieLens platform. The early interactions largely define the user portrait which affect the subsequent interactions. Second, user interactions are highly affected by the candidate movies that are recommended by the platform's internal recommendation algorithm(s). Third, changing the order of user interactions makes it more difficult for sequential algorithms to capture the progressive interaction process. We further discuss the discrepancy between the interaction generation mechanism that is employed by the MovieLens system and that of typical real-world recommendation scenarios. That is, the MovieLens dataset records \(\langle user-MovieLens\rangle\) interactions, but not \(\langle user-movie\rangle\) interactions. All research articles using the MovieLens dataset model the \(\langle user-MovieLens\rangle\) rather than the \(\langle user-movie\rangle\) interactions, making their results less generalizable to many practical recommendation scenarios in real-world settings. In summary, the MovieLens platform demonstrates an efficient and effective way of collecting user preferences to address cold-starts. However, models that achieve excellent recommendation accuracy on the MovieLens dataset may not demonstrate superior performance in practice , for at least two kinds of differences: (1) the differences in the contexts of user-item interaction generation, and (2) the differences in user knowledge about the item collections. While results on MovieLens can be useful as a reference, they should not be solely relied upon as the primary justification for the effectiveness of a recommendation system model.

An Entropy-Based Composite Indicator for Evaluating the Effectiveness of Recommender System Algorithms

The problem of forming a composite indicator for evaluating the effectiveness of recommender system algorithms is considered. A novel composite indicator is proposed by combining individual metrics using the entropy method. The testing base of this study consists of 12 algorithms (on the one hand) and 3 datasets (on the other). For each algorithm–dataset combination, we calculate partial criteria used in evaluating recommender systems. According to the results presented below, the composite indicator is an effective tool for evaluating the performance of recommender system algorithms. As is shown, the performance of the algorithms varies depending on the size and other basic characteristics of a particular dataset. This indicator can be used to develop more efficient algorithms and their ensembles as well as to optimize hyperparameters and improve the quality of recommendations.

Algorithmic Recommender Systems

Abstract Despite their ethical challenges, recommender systems (RS) are widely endorsed as a necessary solution to the problem of information overload. After clarifying how the harmful effects of information overload can be characterized in distinct ways, I explore the often overlooked potential benefits of abundant online spaces. I argue that these spaces afford valuable opportunities to experience spontaneous freedom. I then put forth a more comprehensive evaluation of the role RS should assume in algorithmically structuring the online space. This evaluation aims to reconcile the dual objectives of mitigating the harms of information overload and preserving opportunities to experience spontaneous freedom.

Evaluation of health recommender systems: a scoping review protocol

BackgroundPeople increasingly rely on online health information for their health-related decision-making. Given the overwhelming amount of information available, the risk of misinformation is high. Health recommender systems, which recommend personalised...

Implementation and Evaluation of a Hybrid Recommendation System for the Real Estate Market

Introduction: The real estate market has been transformed by digital technologies, especially Industry 4.0, which has made property searching and evaluation more efficient, improving its accuracy with the use of advanced algorithms. Traditional methods have been replaced by online platforms using modern machine learning (ML) algorithms, leading to the need for personalized recommendation systems to improve user experiences. Methodology: This study designed and implemented a hybrid recommendation system that combines collaborative and content-based filtering techniques. The development process involved four phases: literature review, technology selection, prototype implementation, and system deployment. Findings: The proposed hybrid model effectively addressed challenges such as data sparsity and the cold start problem, improving recommendation accuracy. In the evaluation, users indicated high satisfaction with the system’s ability to offer personalized property recommendations. Conclusion: Thus, hybrid recommendation systems can significantly improve the property search experience by offering personalized recommendations. However, further research into the applicability of the system in different domains remains a need for further exploration.

Design and Evaluation of a Learning Resource Recommendation System Based on Machine Learning

In the era of information overload, learners are often overwhelmed by the vast amount of educational content available online. To address this challenge, we have developed a Learning Resource Recommendation System (LRRS) that leverages machine learning techniques to provide personalized learning material suggestions. The LRRS is designed to analyze learners' profiles, preferences, and learning behaviors to deliver a tailored learning experience. By employing collaborative filtering, content-based filtering, and hybrid methods, the system can predict and recommend resources that are most relevant to individual learners. This paper presents the architecture of the LRRS, the methodologies used for recommendation, and the results of its evaluation. We conducted experiments using a dataset of learners and resources to assess the accuracy and effectiveness of the system. The evaluation metrics, including precision, recall, and F1-score, demonstrate the system's ability to enhance the learning process by providing relevant and engaging content. The LRRS also includes a feedback mechanism that allows for continuous improvement of the recommendation algorithms based on user interactions. The findings of this study contribute to the field of educational technology by offering insights into the development of intelligent systems that can support personalized learning.

Recommender Systems for Teachers: A Systematic Literature Review of Recent (2011–2023) Research

Recommender Systems (RSs) have recently emerged as a practical solution to the information overload problem users face when searching for digital content. In general, RSs provide their respective users with specialized advice and guidance in order to make informed decisions on the selection of suitable digital content. This paper is a systematic literature review of recent (2011–2023) publications on RSs designed and developed in the context of education to support teachers in particular—one of the target groups least frequently addressed by existing RSs. A body of 61 journal papers is selected and analyzed to answer research questions focusing on experimental studies that include RS evaluation and report evaluation results. This review is expected to help teachers in better exploiting RS technology as well as new researchers/developers in this field in better designing and developing RSs for the benefit of teachers. An interesting result obtained through this study is that the recent employment of machine learning algorithms for the generation of recommendations has brought about significant RS quality and performance improvements in terms of recommendation accuracy, personalization and timeliness.

ChatGPT based recommendation system for retail shops

The rapid growth of e-commerce platforms has emphasized the significance of personalized recommendation systems in enhancing user engagement and satisfaction. This research paper presents the development and evaluation of an innovative Product Recommendation System that leverages advanced Artificial Intelligence (AI) techniques to provide tailored product suggestions. The primary objective is to create a user-centric experience by integrating an AI assistant, enabling natural and interactive interactions. Through a comprehensive survey conducted to understand customer behaviour while purchasing product using AI, the study aims to assess the system's effectiveness in delivering accurate recommendations and providing a seamless purchasing experience. The paper contributes to the field by showcasing the practical implementation of AI-driven recommendation systems, highlighting their potential to transform e-commerce interactions.

Socially-Motivated Music Recommendation

Extensive literature spanning psychology, sociology, and musicology has sought to understand the motivations for why people listen to music, including both individually and socially motivated reasons. Music's social functions, while present throughout the world, may be particularly important in collectivist societies, but music recommender systems generally target individualistic functions of music listening. In this study, we explore how a recommender system focused on social motivations for music listening might work by addressing a particular motivation: the desire to listen to music that is trending in one’s community. We frame a recommendation task suited to this desire and propose a corresponding evaluation metric to address the timeliness of recommendations. Using listening data from Spotify, we construct a simple, heuristic-based approach to introduce and explore this recommendation task. Analyzing the effectiveness of this approach, we discuss what we believe is an overlooked trade-off between the precision and timeliness of recommendations, as well as considerations for modeling users' musical communities. Finally, we highlight key cultural differences in the effectiveness of this approach, underscoring the importance of incorporating a diverse cultural perspective in the development and evaluation of recommender systems.

Formalizing Multimedia Recommendation through Multimodal Deep Learning

Recommender systems (RSs) provide customers with a personalized navigation experience within the vast catalogs of products and services offered on popular online platforms. Despite the substantial success of traditional RSs, recommendation remains a highly challenging task, especially in specific scenarios and domains. For example, human affinity for items described through multimedia content (e.g., images, audio, and text), such as fashion products, movies, and music, is multi-faceted and primarily driven by their diverse characteristics. Therefore, by leveraging all available signals in such scenarios, multimodality enables us to tap into richer information sources and construct more refined user/item profiles for recommendations. Despite the growing number of multimodal techniques proposed for multimedia recommendation, the existing literature lacks a shared and universal schema for modeling and solving the recommendation problem through the lens of multimodality. Given the recent advances in multimodal deep learning for other tasks and scenarios where precise theoretical and applicative procedures exist, we also consider it imperative to formalize a general multimodal schema for multimedia recommendation. In this work, we first provide a comprehensive literature review of multimodal approaches for multimedia recommendation from the last eight years. Second, we outline the theoretical foundations of a multimodal pipeline for multimedia recommendation by identifying and formally organizing recurring solutions/patterns; at the same time, we demonstrate its rationale by conceptually applying it to selected state-of-the-art approaches in multimedia recommendation. Third, we conduct a benchmarking analysis of recent algorithms for multimedia recommendation within Elliot, a rigorous framework for evaluating recommender systems, where we re-implement such multimedia recommendation approaches. Finally, we highlight the significant unresolved challenges in multimodal deep learning for multimedia recommendation and suggest possible avenues for addressing them. The primary aim of this work is to provide guidelines for designing and implementing the next generation of multimodal approaches in multimedia recommendation.

The Implementation of Recommender Systems for Mental Health Recovery Narratives: Evaluation of Use and Performance.

Recommender systems help narrow down a large range of items to a smaller, personalized set. NarraGive is a first-in-field hybrid recommender system for mental health recovery narratives, recommending narratives based on their content and narrator characteristics (using content-based filtering) and on narratives beneficially impacting other similar users (using collaborative filtering). NarraGive is integrated into the Narrative Experiences Online (NEON) intervention, a web application providing access to the NEON Collection of recovery narratives. This study aims to analyze the 3 recommender system algorithms used in NarraGive to inform future interventions using recommender systems for lived experience narratives. Using a recently published framework for evaluating recommender systems to structure the analysis, we compared the content-based filtering algorithm and collaborative filtering algorithms by evaluating the accuracy (how close the predicted ratings are to the true ratings), precision (the proportion of the recommended narratives that are relevant), diversity (how diverse the recommended narratives are), coverage (the proportion of all available narratives that can be recommended), and unfairness (whether the algorithms produce less accurate predictions for disadvantaged participants) across gender and ethnicity. We used data from all participants in 2 parallel-group, waitlist control clinical trials of the NEON intervention (NEON trial: N=739; NEON for other [eg, nonpsychosis] mental health problems [NEON-O] trial: N=1023). Both trials included people with self-reported mental health problems who had and had not used statutory mental health services. In addition, NEON trial participants had experienced self-reported psychosis in the previous 5 years. Our evaluation used a database of Likert-scale narrative ratings provided by trial participants in response to validated narrative feedback questions. Participants from the NEON and NEON-O trials provided 2288 and 1896 narrative ratings, respectively. Each rated narrative had a median of 3 ratings and 2 ratings, respectively. For the NEON trial, the content-based filtering algorithm performed better for coverage; the collaborative filtering algorithms performed better for accuracy, diversity, and unfairness across both gender and ethnicity; and neither algorithm performed better for precision. For the NEON-O trial, the content-based filtering algorithm did not perform better on any metric; the collaborative filtering algorithms performed better on accuracy and unfairness across both gender and ethnicity; and neither algorithm performed better for precision, diversity, or coverage. Clinical population may be associated with recommender system performance. Recommender systems are susceptible to a wide range of undesirable biases. Approaches to mitigating these include providing enough initial data for the recommender system (to prevent overfitting), ensuring that items can be accessed outside the recommender system (to prevent a feedback loop between accessed items and recommended items), and encouraging participants to provide feedback on every narrative they interact with (to prevent participants from only providing feedback when they have strong opinions).

Author Correction: A challenge for rounded evaluation of recommender systems

Author Correction: A challenge for rounded evaluation of recommender systems

Introduction to the Special Issue on Perspectives on Recommender Systems Evaluation

Evaluation plays a vital role in recommender systems—in research and practice—whether for confirming algorithmic concepts or assessing the operational validity of designs and applications. It may span the evaluation of early ideas and approaches up to elaborate implementations of systems integrated into everyday product settings; it may target a wide spectrum of different factors being evaluated. In this special issue, we explore recommender systems evaluation—theory and practice—while considering a diverse set of perspectives. These include recommender systems purposes, stakeholders, methodological approaches, and consequences. The collection of articles in this special issue offers insightful analyses of current recommender system evaluation practices, acknowledging their limitations, and setting out future research directions. As recommender systems evolve, the need for adequate evaluation methods and approaches increases. This special issue sheds light on areas undergoing development or requiring added attention from the research and practitioner communities in recommender systems. The compilation serves as a call to the recommender systems research community, motivating continued research and exploration of evaluation metrics, methods, and strategies.

Exploring the Landscape of Recommender Systems Evaluation: Practices and Perspectives

Recommender systems research and practice are fast-developing topics with growing adoption in a wide variety of information access scenarios. In this article, we present an overview of research specifically focused on the evaluation of recommender systems. We perform a systematic literature review, in which we analyze 57 papers spanning six years (2017–2022). Focusing on the processes surrounding evaluation, we dial in on the methods applied, the datasets utilized, and the metrics used. Our study shows that the predominant experiment type in research on the evaluation of recommender systems is offline experimentation and that online evaluations are primarily used in combination with other experimentation methods, e.g., an offline experiment. Furthermore, we find that only a few datasets (MovieLens, Amazon review dataset) are widely used, while many datasets are used in only a few papers each. We observe a similar scenario when analyzing the employed performance metrics—a few metrics are widely used (precision, normalized Discounted Cumulative Gain, and Recall), while many others are used in only a few papers. Overall, our review indicates that beyond-accuracy qualities are rarely assessed. Our analysis shows that the research community working on evaluation has focused on the development of evaluation in a rather narrow scope, with the majority of experiments focusing on a few metrics, datasets, and methods.

Distributionally-Informed Recommender System Evaluation

Current practice for evaluating recommender systems typically focuses on point estimates of user-oriented effectiveness metrics or business metrics, sometimes combined with additional metrics for considerations such as diversity and novelty. In this article, we argue for the need for researchers and practitioners to attend more closely to various distributions that arise from a recommender system (or other information access system) and the sources of uncertainty that lead to these distributions. One immediate implication of our argument is that both researchers and practitioners must report and examine more thoroughly the distribution of utility between and within different stakeholder groups. However, distributions of various forms arise in many more aspects of the recommender systems experimental process, and distributional thinking has substantial ramifications for how we design, evaluate, and present recommender systems evaluation and research results. Leveraging and emphasizing distributions in the evaluation of recommender systems is a necessary step to ensure that the systems provide appropriate and equitably distributed benefit to the people they affect.

Towards Simulation-Based Evaluation of Recommender Systems with Carousel Interfaces

Offline data-driven evaluation is considered a low-cost and more accessible alternative to the online empirical method of assessing the quality of recommender systems. Despite their popularity and effectiveness, most data-driven approaches are unsuitable for evaluating interactive recommender systems. In this article, we attempt to address this issue by simulating the user interactions with the system as a part of the evaluation process. Particularly, we demonstrate that simulated users find their desired item more efficiently when recommendations are presented as a list of carousels compared to a simple ranked list.

Group Validation in Recommender Systems: Framework for Multi-layer Performance Evaluation

Evaluation of recommendation systems continues evolving, especially in recent years. There have been several attempts to standardize the assessment processes and propose replacement metrics better oriented toward measuring effective personalization. However, standard evaluation tools merely possess the capacity to provide a general overview of a system’s performance; they lack consistency and effectiveness in their use, as evidenced by most recent studies on the topic. Furthermore, traditional evaluation techniques fail to detect potentially harmful data on small subsets. Moreover, they generally lack explainable features to interpret how such minor variations could affect the system’s performance. This proposal focuses on data clustering for recommender evaluation and applies a cluster assessment technique to locate such performance issues. Our new approach, namedgroup validation, aids in spotting critical performance variability in compact subsets of the system’s data and unravels hidden weaknesses in predictions where such unfavorable variations generally go unnoticed with typical assessment methods. Group validation for recommenders is a modular evaluation layer that complements regular evaluation and includes a new unique perspective to the evaluation process. Additionally, it allows several applications to the recommender ecosystem, such as model evolution tests, fraud/attack detection, and the capacity for hosting a hybrid model setup.

A Framework and Toolkit for Testing the Correctness of Recommendation Algorithms

Evaluating recommender systems adequately and thoroughly is an important task. Significant efforts are dedicated to proposing metrics, methods, and protocols for doing so. However, there has been little discussion in the recommender systems’ literature on the topic of testing. In this work, we adopt and adapt concepts from the software testing domain, e.g., code coverage, metamorphic testing, or property-based testing, to help researchers to detect and correct faults in recommendation algorithms. We propose a test suite that can be used to validate the correctness of a recommendation algorithm, and thus identify and correct issues that can affect the performance and behavior of these algorithms. Our test suite contains both black box and white box tests at every level of abstraction, i.e., system, integration, and unit. To facilitate adoption, we release RecPack Tests , an open-source Python package containing template test implementations. We use it to test four popular Python packages for recommender systems: RecPack , PyLensKit , Surprise , and Cornac . Despite the high test coverage of each of these packages, we find that we are still able to uncover undocumented functional requirements and even some bugs. This validates our thesis that testing the correctness of recommendation algorithms can complement traditional methods for evaluating recommendation algorithms.

A Comprehensive Survey on Recommender Systems Techniques and Challenges in Big Data Analytics with IOT Applications

Purpose: Purpose of this research is to carry out survey on Recommendation systems techniques in Big Data Analytics. This article presents designing of recommender systems and evaluates it with help of various performance metrics in IoT applications.
 
 Theoretical framework: With fast development and applications of Internet of Things, large amount of user data is generated and accumulated every day. Growth of media consumption in online social networks is exponential which requires an efficient and effective recommendation system to enhance excellence in experience for users. Recommender systems help users to overcome Information Overload problem by providing them relevant contents. 
 
 Method/design/approach: The main aspect of recommender system is how to take complete advantage of this ubiquitous data. Recommender system is mainly used to guess or predict users’ interests and make relevant recommendations. Collaborative filtering is the technique that uses the relationships between users and between items in order to build a prediction. Collaborative filtering algorithms are mainly categorized as model-based methods and memory-based methods. In this article, various methods to build recommender system are described. Similarly, Collaborative filtering uses Pearson cosine, cosine vector, Jaccard similarity to identify same users or items. Recommender system has various applications in domain such as healthcare, transportation, agriculture, e-media etc. 
 
 Findings: Evaluation of recommender system with help of metrics such as Precision and Recall is presented. Comparison of experimental results is presented with help of MAE and RMSE. Recommendation system helps to discover relevant insights and can be one of the vital technologies in future IoT solutions.
 
 Research, Practical & social implications: The research makes significant contribution by providing survey of existing recommender systems along with challenges faced while designing effective and accurate recommender. Various similarity measures to find similar users or items are investigated with future pointer direction. Recommender system help in decision making process.
 
 Originality/value: The results and conclusion obtained in this research are helpful in development of novel Recommender systems which definitely assist users to overcome Information Overload issue. It helps user to save network load as well.

Sistem Rekomendasi Film Menggunakan Metode K-NN

Recommendation systems are at the heart of cutting through the noise of online data, helping users find content that matches their preferences. In this regard, the K-Nearest Neighbors (KNN) method stands out as a promising approach. KNN, as a similarity-based algorithm, utilizes information from nearest neighbors to make predictions or recommendations. This research explores the implementation of KNN in developing a film recommendation system with a focus on increasing the accuracy and relevance of recommendations. Related references, such as the evaluation of recommendation systems by Herlocker et al. (2004) and the concept of collaborative filtering by Resnick and Varian (1997), are the basis for understanding and improving the potential of KNN. By detailing the exploration of this concept, it is hoped that this research will provide a comprehensive and holistic view in the development of a film recommendation system. From testing using the performance test of the KNN method, namely accuracy, recall, and precision with the best value of 45.4% for accuracy, 45.4% for recall, and 100% precision so that the algorithm can be applied in the film recommendation system.