Evaluation Of Recommender Systems Research Articles

Formalizing Multimedia Recommendation through Multimodal Deep Learning

Recommender systems (RSs) provide customers with a personalized navigation experience within the vast catalogs of products and services offered on popular online platforms. Despite the substantial success of traditional RSs, recommendation remains a highly challenging task, especially in specific scenarios and domains. For example, human affinity for items described through multimedia content (e.g., images, audio, and text), such as fashion products, movies, and music, is multi-faceted and primarily driven by their diverse characteristics. Therefore, by leveraging all available signals in such scenarios, multimodality enables us to tap into richer information sources and construct more refined user/item profiles for recommendations. Despite the growing number of multimodal techniques proposed for multimedia recommendation, the existing literature lacks a shared and universal schema for modeling and solving the recommendation problem through the lens of multimodality. Given the recent advances in multimodal deep learning for other tasks and scenarios where precise theoretical and applicative procedures exist, we also consider it imperative to formalize a general multimodal schema for multimedia recommendation. In this work, we first provide a comprehensive literature review of multimodal approaches for multimedia recommendation from the last eight years. Second, we outline the theoretical foundations of a multimodal pipeline for multimedia recommendation by identifying and formally organizing recurring solutions/patterns; at the same time, we demonstrate its rationale by conceptually applying it to selected state-of-the-art approaches in multimedia recommendation. Third, we conduct a benchmarking analysis of recent algorithms for multimedia recommendation within Elliot, a rigorous framework for evaluating recommender systems, where we re-implement such multimedia recommendation approaches. Finally, we highlight the significant unresolved challenges in multimodal deep learning for multimedia recommendation and suggest possible avenues for addressing them. The primary aim of this work is to provide guidelines for designing and implementing the next generation of multimodal approaches in multimedia recommendation.

The Implementation of Recommender Systems for Mental Health Recovery Narratives: Evaluation of Use and Performance.

Recommender systems help narrow down a large range of items to a smaller, personalized set. NarraGive is a first-in-field hybrid recommender system for mental health recovery narratives, recommending narratives based on their content and narrator characteristics (using content-based filtering) and on narratives beneficially impacting other similar users (using collaborative filtering). NarraGive is integrated into the Narrative Experiences Online (NEON) intervention, a web application providing access to the NEON Collection of recovery narratives. This study aims to analyze the 3 recommender system algorithms used in NarraGive to inform future interventions using recommender systems for lived experience narratives. Using a recently published framework for evaluating recommender systems to structure the analysis, we compared the content-based filtering algorithm and collaborative filtering algorithms by evaluating the accuracy (how close the predicted ratings are to the true ratings), precision (the proportion of the recommended narratives that are relevant), diversity (how diverse the recommended narratives are), coverage (the proportion of all available narratives that can be recommended), and unfairness (whether the algorithms produce less accurate predictions for disadvantaged participants) across gender and ethnicity. We used data from all participants in 2 parallel-group, waitlist control clinical trials of the NEON intervention (NEON trial: N=739; NEON for other [eg, nonpsychosis] mental health problems [NEON-O] trial: N=1023). Both trials included people with self-reported mental health problems who had and had not used statutory mental health services. In addition, NEON trial participants had experienced self-reported psychosis in the previous 5 years. Our evaluation used a database of Likert-scale narrative ratings provided by trial participants in response to validated narrative feedback questions. Participants from the NEON and NEON-O trials provided 2288 and 1896 narrative ratings, respectively. Each rated narrative had a median of 3 ratings and 2 ratings, respectively. For the NEON trial, the content-based filtering algorithm performed better for coverage; the collaborative filtering algorithms performed better for accuracy, diversity, and unfairness across both gender and ethnicity; and neither algorithm performed better for precision. For the NEON-O trial, the content-based filtering algorithm did not perform better on any metric; the collaborative filtering algorithms performed better on accuracy and unfairness across both gender and ethnicity; and neither algorithm performed better for precision, diversity, or coverage. Clinical population may be associated with recommender system performance. Recommender systems are susceptible to a wide range of undesirable biases. Approaches to mitigating these include providing enough initial data for the recommender system (to prevent overfitting), ensuring that items can be accessed outside the recommender system (to prevent a feedback loop between accessed items and recommended items), and encouraging participants to provide feedback on every narrative they interact with (to prevent participants from only providing feedback when they have strong opinions).

Author Correction: A challenge for rounded evaluation of recommender systems

Author Correction: A challenge for rounded evaluation of recommender systems

Introduction to the Special Issue on Perspectives on Recommender Systems Evaluation

Evaluation plays a vital role in recommender systems—in research and practice—whether for confirming algorithmic concepts or assessing the operational validity of designs and applications. It may span the evaluation of early ideas and approaches up to elaborate implementations of systems integrated into everyday product settings; it may target a wide spectrum of different factors being evaluated. In this special issue, we explore recommender systems evaluation—theory and practice—while considering a diverse set of perspectives. These include recommender systems purposes, stakeholders, methodological approaches, and consequences. The collection of articles in this special issue offers insightful analyses of current recommender system evaluation practices, acknowledging their limitations, and setting out future research directions. As recommender systems evolve, the need for adequate evaluation methods and approaches increases. This special issue sheds light on areas undergoing development or requiring added attention from the research and practitioner communities in recommender systems. The compilation serves as a call to the recommender systems research community, motivating continued research and exploration of evaluation metrics, methods, and strategies.

Exploring the Landscape of Recommender Systems Evaluation: Practices and Perspectives

Recommender systems research and practice are fast-developing topics with growing adoption in a wide variety of information access scenarios. In this article, we present an overview of research specifically focused on the evaluation of recommender systems. We perform a systematic literature review, in which we analyze 57 papers spanning six years (2017–2022). Focusing on the processes surrounding evaluation, we dial in on the methods applied, the datasets utilized, and the metrics used. Our study shows that the predominant experiment type in research on the evaluation of recommender systems is offline experimentation and that online evaluations are primarily used in combination with other experimentation methods, e.g., an offline experiment. Furthermore, we find that only a few datasets (MovieLens, Amazon review dataset) are widely used, while many datasets are used in only a few papers each. We observe a similar scenario when analyzing the employed performance metrics—a few metrics are widely used (precision, normalized Discounted Cumulative Gain, and Recall), while many others are used in only a few papers. Overall, our review indicates that beyond-accuracy qualities are rarely assessed. Our analysis shows that the research community working on evaluation has focused on the development of evaluation in a rather narrow scope, with the majority of experiments focusing on a few metrics, datasets, and methods.

Distributionally-Informed Recommender System Evaluation

Current practice for evaluating recommender systems typically focuses on point estimates of user-oriented effectiveness metrics or business metrics, sometimes combined with additional metrics for considerations such as diversity and novelty. In this article, we argue for the need for researchers and practitioners to attend more closely to various distributions that arise from a recommender system (or other information access system) and the sources of uncertainty that lead to these distributions. One immediate implication of our argument is that both researchers and practitioners must report and examine more thoroughly the distribution of utility between and within different stakeholder groups. However, distributions of various forms arise in many more aspects of the recommender systems experimental process, and distributional thinking has substantial ramifications for how we design, evaluate, and present recommender systems evaluation and research results. Leveraging and emphasizing distributions in the evaluation of recommender systems is a necessary step to ensure that the systems provide appropriate and equitably distributed benefit to the people they affect.

Towards Simulation-Based Evaluation of Recommender Systems with Carousel Interfaces

Offline data-driven evaluation is considered a low-cost and more accessible alternative to the online empirical method of assessing the quality of recommender systems. Despite their popularity and effectiveness, most data-driven approaches are unsuitable for evaluating interactive recommender systems. In this article, we attempt to address this issue by simulating the user interactions with the system as a part of the evaluation process. Particularly, we demonstrate that simulated users find their desired item more efficiently when recommendations are presented as a list of carousels compared to a simple ranked list.

Group Validation in Recommender Systems: Framework for Multi-layer Performance Evaluation

Evaluation of recommendation systems continues evolving, especially in recent years. There have been several attempts to standardize the assessment processes and propose replacement metrics better oriented toward measuring effective personalization. However, standard evaluation tools merely possess the capacity to provide a general overview of a system’s performance; they lack consistency and effectiveness in their use, as evidenced by most recent studies on the topic. Furthermore, traditional evaluation techniques fail to detect potentially harmful data on small subsets. Moreover, they generally lack explainable features to interpret how such minor variations could affect the system’s performance. This proposal focuses on data clustering for recommender evaluation and applies a cluster assessment technique to locate such performance issues. Our new approach, named group validation , aids in spotting critical performance variability in compact subsets of the system’s data and unravels hidden weaknesses in predictions where such unfavorable variations generally go unnoticed with typical assessment methods. Group validation for recommenders is a modular evaluation layer that complements regular evaluation and includes a new unique perspective to the evaluation process. Additionally, it allows several applications to the recommender ecosystem, such as model evolution tests, fraud/attack detection, and the capacity for hosting a hybrid model setup.

A Framework and Toolkit for Testing the Correctness of Recommendation Algorithms

Evaluating recommender systems adequately and thoroughly is an important task. Significant efforts are dedicated to proposing metrics, methods, and protocols for doing so. However, there has been little discussion in the recommender systems’ literature on the topic of testing. In this work, we adopt and adapt concepts from the software testing domain, e.g., code coverage, metamorphic testing, or property-based testing, to help researchers to detect and correct faults in recommendation algorithms. We propose a test suite that can be used to validate the correctness of a recommendation algorithm, and thus identify and correct issues that can affect the performance and behavior of these algorithms. Our test suite contains both black box and white box tests at every level of abstraction, i.e., system, integration, and unit. To facilitate adoption, we release RecPack Tests , an open-source Python package containing template test implementations. We use it to test four popular Python packages for recommender systems: RecPack , PyLensKit , Surprise , and Cornac . Despite the high test coverage of each of these packages, we find that we are still able to uncover undocumented functional requirements and even some bugs. This validates our thesis that testing the correctness of recommendation algorithms can complement traditional methods for evaluating recommendation algorithms.

A Comprehensive Survey on Recommender Systems Techniques and Challenges in Big Data Analytics with IOT Applications

Purpose: Purpose of this research is to carry out survey on Recommendation systems techniques in Big Data Analytics. This article presents designing of recommender systems and evaluates it with help of various performance metrics in IoT applications.
 
 Theoretical framework: With fast development and applications of Internet of Things, large amount of user data is generated and accumulated every day. Growth of media consumption in online social networks is exponential which requires an efficient and effective recommendation system to enhance excellence in experience for users. Recommender systems help users to overcome Information Overload problem by providing them relevant contents. 
 
 Method/design/approach: The main aspect of recommender system is how to take complete advantage of this ubiquitous data. Recommender system is mainly used to guess or predict users’ interests and make relevant recommendations. Collaborative filtering is the technique that uses the relationships between users and between items in order to build a prediction. Collaborative filtering algorithms are mainly categorized as model-based methods and memory-based methods. In this article, various methods to build recommender system are described. Similarly, Collaborative filtering uses Pearson cosine, cosine vector, Jaccard similarity to identify same users or items. Recommender system has various applications in domain such as healthcare, transportation, agriculture, e-media etc. 
 
 Findings: Evaluation of recommender system with help of metrics such as Precision and Recall is presented. Comparison of experimental results is presented with help of MAE and RMSE. Recommendation system helps to discover relevant insights and can be one of the vital technologies in future IoT solutions.
 
 Research, Practical & social implications: The research makes significant contribution by providing survey of existing recommender systems along with challenges faced while designing effective and accurate recommender. Various similarity measures to find similar users or items are investigated with future pointer direction. Recommender system help in decision making process.
 
 Originality/value: The results and conclusion obtained in this research are helpful in development of novel Recommender systems which definitely assist users to overcome Information Overload issue. It helps user to save network load as well.

Sistem Rekomendasi Film Menggunakan Metode K-NN

Recommendation systems are at the heart of cutting through the noise of online data, helping users find content that matches their preferences. In this regard, the K-Nearest Neighbors (KNN) method stands out as a promising approach. KNN, as a similarity-based algorithm, utilizes information from nearest neighbors to make predictions or recommendations. This research explores the implementation of KNN in developing a film recommendation system with a focus on increasing the accuracy and relevance of recommendations. Related references, such as the evaluation of recommendation systems by Herlocker et al. (2004) and the concept of collaborative filtering by Resnick and Varian (1997), are the basis for understanding and improving the potential of KNN. By detailing the exploration of this concept, it is hoped that this research will provide a comprehensive and holistic view in the development of a film recommendation system. From testing using the performance test of the KNN method, namely accuracy, recall, and precision with the best value of 45.4% for accuracy, 45.4% for recall, and 100% precision so that the algorithm can be applied in the film recommendation system.

Beyond-accuracy: a review on diversity, serendipity, and fairness in recommender systems based on graph neural networks.

By providing personalized suggestions to users, recommender systems have become essential to numerous online platforms. Collaborative filtering, particularly graph-based approaches using Graph Neural Networks (GNNs), have demonstrated great results in terms of recommendation accuracy. However, accuracy may not always be the most important criterion for evaluating recommender systems' performance, since beyond-accuracy aspects such as recommendation diversity, serendipity, and fairness can strongly influence user engagement and satisfaction. This review paper focuses on addressing these dimensions in GNN-based recommender systems, going beyond the conventional accuracy-centric perspective. We begin by reviewing recent developments in approaches that improve not only the accuracy-diversity trade-off but also promote serendipity, and fairness in GNN-based recommender systems. We discuss different stages of model development including data preprocessing, graph construction, embedding initialization, propagation layers, embedding fusion, score computation, and training methodologies. Furthermore, we present a look into the practical difficulties encountered in assuring diversity, serendipity, and fairness, while retaining high accuracy. Finally, we discuss potential future research directions for developing more robust GNN-based recommender systems that go beyond the unidimensional perspective of focusing solely on accuracy. This review aims to provide researchers and practitioners with an in-depth understanding of the multifaceted issues that arise when designing GNN-based recommender systems, setting our work apart by offering a comprehensive exploration of beyond-accuracy dimensions.

Report on the 3rd Workshop on the Perspectives on the Evaluation of Recommender Systems (PERSPECTIVES 2023) at RecSys 2023

Evaluation is a central step when developing, optimizing, and deploying recommender systems. The PERSPECTIVES 2023 workshop, held as part of the 17th ACM Conference on Recommender Systems (RecSys 2023), served as a forum where researchers from both academia and industry critically reflected on the evaluation of recommender systems. The goal of the PERSPECTIVES workshop series is to capture the current state of evaluation from different perspectives and discuss the different targets that recommender systems evaluation should strive for. In the third edition of the workshop, we discussed problems and lessons learned, and aimed to move the discourse forward within the community. Date : 19 September 2023. Website : https://perspectives-ws.github.io/2023/.

A Comprehensive Survey on Recommender Systems Techniques and Challenges in Big Data Analytics with IoT Applications

Purpose: Purpose of this research is to carry out survey on Recommendation systems techniques in Big Data Analytics. This article presents designing of recommender systems and evaluates it with help of various performance metrics in IoT applications.
 
 Theoretical framework: With fast development and applications of Internet of Things, large amount of user data is generated and accumulated every day. Growth of media consumption in online social networks is exponential which requires an efficient and effective recommendation system to enhance excellence in experience for users. Recommender systems help users to overcome Information Overload problem by providing them relevant contents. 
 
 Method/design/approach: The main aspect of recommender system is how to take complete advantage of this ubiquitous data. Recommender system is mainly used to guess or predict users’ interests and make relevant recommendations. Collaborative filtering is the technique that uses the relationships between users and between items in order to build a prediction. Collaborative filtering algorithms are mainly categorized as model-based methods and memory-based methods. In this article, various methods to build recommender system are described. Similarly, Collaborative filtering uses Pearson cosine, cosine vector, Jaccard similarity to identify same users or items. Recommender system has various applications in domain such as healthcare, transportation, agriculture, e-media etc. 
 
 Findings: Evaluation of recommender system with help of metrics such as Precision and Recall is presented. Comparison of experimental results is presented with help of MAE and RMSE. Recommendation system helps to discover relevant insights and can be one of the vital technologies in future IoT solutions.
 
 Research, Practical & social implications: The research makes significant contribution by providing survey of existing recommender systems along with challenges faced while designing effective and accurate recommender. Various similarity measures to find similar users or items are investigated with future pointer direction. Recommender system help in decision making process.
 
 Originality/value: The results and conclusion obtained in this research are helpful in development of novel Recommender systems which definitely assist users to overcome Information Overload issue. It helps user to save network load as well.

Usability Evaluation of a Knowledge Graph-Based Dementia Care Intelligent Recommender System: Mixed Methods Study.

Knowledge graph-based recommender systems offer the possibility of meeting the personalized needs of people with dementia and their caregivers. However, the usability of such a recommender system remains unknown. This study aimed to evaluate the usability of a knowledge graph-based dementia care intelligent recommender system (DCIRS). We used a convergent mixed methods design to conduct the usability evaluation, including the collection of quantitative and qualitative data. Participants were recruited through social media advertisements. After 2 weeks of DCIRS use, feedback was collected with the Computer System Usability Questionnaire and semistructured interviews. Descriptive statistics were used to describe sociodemographic characteristics and questionnaire scores. Qualitative data were analyzed systematically using inductive thematic analysis. A total of 56 caregivers were recruited. Quantitative data suggested that the DCIRS was easy for caregivers to use, and the mean questionnaire score was 2.14. Qualitative data showed that caregivers generally believed that the content of the DCIRS was professional, easy to understand, and instructive, and could meet users' personalized needs; they were willing to continue to use it. However, the DCIRS also had some shortcomings. Functions that enable interactions between professionals and caregivers and that provide caregiver support and resource recommendations might be added to improve the system's usability. The recommender system provides a solution to meet the personalized needs of people with dementia and their caregivers and has the potential to substantially improve health outcomes. The next step will be to optimize and update the recommender system based on caregivers' suggestions and evaluate the effect of the application.

A Reusable Model-agnostic Framework for Faithfully Explainable Recommendation and System Scrutability

State-of-the-art industrial-level recommender system applications mostly adopt complicated model structures such as deep neural networks. While this helps with the model performance, the lack of system explainability caused by these nearly blackbox models also raises concerns and potentially weakens the users’ trust in the system. Existing work on explainable recommendation mostly focuses on designing interpretable model structures to generate model-intrinsic explanations. However, most of them have complex structures, and it is difficult to directly apply these designs onto existing recommendation applications due to the effectiveness and efficiency concerns. However, while there have been some studies on explaining recommendation models without knowing their internal structures (i.e., model-agnostic explanations), these methods have been criticized for not reflecting the actual reasoning process of the recommendation model or, in other words,faithfulness. How to develop model-agnostic explanation methods and evaluate them in terms of faithfulness is mostly unknown. In this work, we propose a reusable evaluation pipeline for model-agnostic explainable recommendation. Our pipeline evaluates the quality of model-agnostic explanation from the perspectives of faithfulness and scrutability. We further propose a model-agnostic explanation framework for recommendation and verify it with the proposed evaluation pipeline. Extensive experiments on public datasets demonstrate that our model-agnostic framework is able to generate explanations that are faithful to the recommendation model. We additionally provide quantitative and qualitative study to show that our explanation framework could enhance the scrutability of blackbox recommendation model. With proper modification, our evaluation pipeline and model-agnostic explanation framework could be easily migrated to existing applications. Through this work, we hope to encourage the community to focus more on faithfulness evaluation of explainable recommender systems.

Fairness in Recommender Systems: Evaluation Approaches and Assurance Strategies

With the wide application of recommender systems, the potential impacts of recommender systems on customers, item providers and other parties have attracted increasing attention. Fairness, which is the quality of treating people equally, is also becoming important in recommender system evaluation and algorithm design. Therefore, in the past years, there has been a growing interest in fairness measurement and assurance in recommender systems. Although there are several reviews on related topics, such as fairness in machine learning and debias in recommender systems, they do not present a systematic view on fairness in recommender systems, which is context aware and has a multi-sided meaning. Therefore, in this review, the concept of fairness is discussed in detail in the various contexts of recommender systems. Specifically, a comprehensive framework to classify fairness metrics is proposed from four dimensions, i.e.,Fairness for Whom,Demographic Unit,Time Frame, andQuantification Method. Then the strategies for eliminating unfairness in recommendations, fairness in different recommendation tasks and datasets are reviewed and summarized. Finally, the challenges and future work are discussed.

Towards Reliable Item Sampling for Recommendation Evaluation

Since Rendle and Krichene argued that commonly used sampling-based evaluation metrics are ``inconsistent'' with respect to the global metrics (even in expectation), there have been a few studies on the sampling-based recommender system evaluation. Existing methods try either mapping the sampling-based metrics to their global counterparts or more generally, learning the empirical rank distribution to estimate the top-K metrics. However, despite existing efforts, there is still a lack of rigorous theoretical understanding of the proposed metric estimators, and the basic item sampling also suffers from the ``blind spot'' issue, i.e., estimation accuracy to recover the top-K metrics when K is small can still be rather substantial. In this paper, we provide an in-depth investigation into these problems and make two innovative contributions. First, we propose a new item-sampling estimator that explicitly optimizes the error with respect to the ground truth, and theoretically highlights its subtle difference against prior work. Second, we propose a new adaptive sampling method that aims to deal with the ``blind spot'' problem and also demonstrate the expectation-maximization (EM) algorithm can be generalized for such a setting. Our experimental results confirm our statistical analysis and the superiority of the proposed works. This study helps lay the theoretical foundation for adopting item sampling metrics for recommendation evaluation and provides strong evidence for making item sampling a powerful and reliable tool for recommendation evaluation.

Overview of definition, evaluation, and algorithms of serendipity in recommender systems

Over time, recommendation systems are playing an important role in an increasingly wide range of areas, such as paper retrieval sites that can recommend papers or books to users, and shopping sites that can recommend products to users. With the development of recommendation systems, there are many different metrics to measure a good recommendation system, including serendipity. This paper summarizes the definition of serendipity, a review of the metrics for measuring serendipity, and several major serendipity-oriented algorithms and presents conjectures for future research on serendipity. Through the research of some papers, for how to delimit and evaluate recommender systems, experts have mostly focused on the unexpected, and most of them use and optimize collaborative filtering algorithms to achieve and improve serendipity.

Recommender System Metaheuristic for Optimizing Decision-Making Computation

We implement a novel recommender system (RS) metaheuristic framework within a nonlinear NP-hard decision-making problem, for reducing the solution search space before high-burden computational steps are performed. Our RS-based metaheuristic supports consideration of comprehensive evaluation criteria, including estimations of the potential solution set’s optimality, diversity, and feedback/preference of the end-user, while also being fully compatible with additional established RS evaluation metrics. Compared to prior Operations Research metaheuristics, our RS-based metaheuristic allows for (1) achieving near-optimal solution scores through comprehensive deep learning training, (2) fast metaheuristic parameter inference during solution instantiation trials, and (3) the ability to reuse this trained RS module for traditional RS ranking of final solution options for the end-user. When implementing this RS metaheuristic within an experimental high-dimensionality simulation environment, we see an average 91.7% reduction in computation time against a baseline approach, and solution scores within 9.1% of theoretical optimal scores. A simplified RS metaheuristic technique was also developed in a more realistic decision-making environment dealing with multidomain command and control scenarios, where a significant computation time reduction of 87.5% is also achieved compared with a baseline approach, while maintaining solution scores within 9.5% of theoretical optimal scores.