Randomized latent factor model for high-dimensional and sparse matrices from industrial applications

Latent factor (LF) models are highly efficient in recommender systems. The problem of LF analysis is defined on high-dimensional and sparse (HiDS) matrices corresponding to relationships among numerous entities in industrial applications. It is ill-posed without a unique and optimal solution. Hence, regularization schemes are vital in improving the generality of an LF model. This work innovatively applies the elastic net-based regularization to an LF model defined on HiDS matrices. To do so, we have 1) adapted the elastic net-based regularization scheme to an LF model to fit the sparsity of an HiDS matrix; and 2) designed feasible and efficient algorithm for an LF model with elastic net-based regularization. Experimental results on two large, real datasets show that with properly-tuned and elastic net-based regularization, the resultant model achieves 1) high prediction accuracy for missing data in an HiDS matrix; 2) high computational efficiency; and 3) sparse LF distribution.

Latent factor (LF) models have proven to be accurate and efficient in extracting hidden knowledge from high-dimensional and sparse (HiDS) matrices. However, most LF models fail to fulfill the non-negativity constraints that reflect the non-negative nature of industrial data. Yet existing non-negative LF models for HiDS matrices suffer from slow convergence leading to considerable time cost. An alternating direction method-based non-negative latent factor (ANLF) model decomposes a non-negative optimization process into small sub-tasks. It updates each LF non-negatively based on the latest state of those trained before, thereby achieving fast convergence and maintaining high prediction accuracy and scalability. This paper theoretically analyze the characteristics of an ANLF model, and presents detailed empirical study regarding its performance on several HiDS matrices arising from industrial applications currently in use. Therefore, its capability of addressing HiDS matrices is validated in both theory and practice.

High-dimensional and sparse (HiDS) matrices generated by recommender systems (RSs) contain rich knowledge. A latent factor (LF) model can address such data effectively. Stochastic gradient descent (SGD) is an efficient algorithm for building a LF model on an HiDS matrix. However, it suffers slow convergence. To address this issue, this study proposes to implement a LF model with a proportional integral derivative (PID) controller. The main idea is to continuously apply a correction for SGD to accelerate the training process. Based on such design, a PID-based LF (PLF) model is proposed. Empirical studies on two HiDS matrices from RSs indicate that a PLF model outperforms an LF model in terms of both convergence rate and prediction accuracy for missing data.

High-dimensional and sparse (HiDS) matrices are commonly seen in big-data-related industrial applications like recommender systems. Latent factor (LF) models have proven to be accurate and efficient in extracting hidden knowledge from them. However, they mostly fail to fulfill the non-negativity constraints that describe the non-negative nature of many industrial data. Moreover, existing models suffer from slow convergence rate. An alternating-direction-method of multipliers-based non-negative LF (AMNLF) model decomposes the task of non-negative LF analysis on an HiDS matrix into small subtasks, where each task is solved based on the latest solutions to the previously solved ones, thereby achieving fast convergence and high prediction accuracy for its missing data. This paper theoretically analyzes the characteristics of an AMNLF model, and presents detailed empirical studies regarding its performance on nine HiDS matrices from industrial applications currently in use. Therefore, its capability of addressing HiDS matrices is justified in both theory and practice.

Non-negative latent factor (NLF) models can efficiently acquire useful knowledge from high-dimensional and sparse (HiDS) matrices filled with non-negative data. Single latent factor-dependent, non-negative and multiplicative update (SLF-NMU) is an efficient algorithm for building an NLF model on an HiDS matrix, yet it suffers slow convergence. On the other hand, a momentum method is frequently adopted to accelerate a learning algorithm explicitly depending on gradients, yet it is incompatible with learning algorithms implicitly depending on gradients, like SLF-NMU. To build a fast NLF model, we firstly propose a generalized momentum method compatible with SLF-NMU. With it, we propose the single latent factor-dependent, non-negative, multiplicative and momentum-integrated update (SLF-NM <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> U) algorithm for accelerating the building process of an NLF model, thereby achieving a fast non-negative latent factor (FNLF) model. Empirical studies on six HiDS matrices from industrial application indicate that with the incorporated momentum effects, FNLF outperforms NLF in terms of both convergence rate and prediction accuracy for missing data. Hence, compare with an NLF model, an FNLF model is more practical in industrial applications.

Non-negativity is vital for latent factor models to preserve the important feature of most high-dimensional and sparse (HiDS) matrices, e.g., none of their entries is negative. With the consideration of non-negativity, the training process of a latent factor model should be specified to achieve constraints-incorporated learning schemes. However, such schemes are neither flexible nor extensible. This work investigates algorithms of inherently non-negative latent factor analysis, which separates non-negativity constraints from the training process. Based on a deep investigation into the learning objective of a non-negative latent factor model, we separate the desired latent factors from decision variables involved in the training process via a single-element-dependent mapping function that makes the output factors inherently non-negative. Then we theoretically prove that the resultant model is able to represent the original one effectively. As a result, we design a highly efficient algorithm to bring the Inherently Non-negative Latent Factor model into practice. Experimental results on three HiDS matrices from industrial recommender systems show that compared with state-of-the-art non-negative latent factor models, the proposed one is able to obtain advantage in prediction accuracy with comparable or higher computational efficiency. Moreover, such high performance is achieved through its unconstrained optimization process on the premise of fulfilling the non-negativity constraints. Hence, the proposed model is highly valuable for industrial applications required to handle HiDS matrices subject to non-negativity constraints.

Recommender systems (RSs) commonly describe its user-item preferences with a high-dimensional and sparse (HiDS) matrix filled with non-negative data. A non-negative latent factor (NLF) model relying on a single latent factor-dependent, non-negative and multiplicative update (SLF-NMU) algorithm is frequently adopted to process such an HiDS matrix. However, an NLF model mostly adopts Euclidean distance for its objective function, which is naturally a special case of α-β-divergence. Moreover, it frequently suffers slow convergence. For addressing these issues, this study proposes a generalized and fast-converging non-negative latent factor (GFNLF) model. Its main idea is two-fold: a) adopting α-β-divergence for its objective function, thereby enhancing its representation ability for HiDS data; b) deducing its momentum-incorporated non-negative multiplicative update (MNMU) algorithm, thereby achieving its fast convergence. Empirical studies on two HiDS matrices emerging from real RSs demonstrate that with carefully-tuned hyperparameters, a GFNLF model outperforms state-of-the-art models in both computational efficiency and prediction accuracy for missing data of an HiDS matrix.

National Key Research and Development Program of China (Grant Number: 2017YFC0804002); 10.13039/501100001809-National Natural Science Foundation of China (Grant Number: 61772493 and 91646114); Chongqing Research Program of Technology Innovation and Application (Grant Number: cstc2017rgzn-zdyfX0020, cstc2017zdcy-zdyf0554 and cstc2017rgzn-zdyf0118); Chongqing Cultivation Program of Innovation and Entrepreneurship Demonstration Group (Grant Number: cstc2017kjrc-cxcytd0149); Chongqing Overseas Scholars Innovation Program (Grant Number: cx2017012 and cx2018011); 10.13039/501100002367-Pioneer Hundred Talents Program of Chinese Academy of Sciences.

To quantify user-item preferences, a recommender system (RS) commonly adopts a high-dimensional and sparse (HiDS) matrix. Such a matrix can be represented by a non-negative latent factor analysis model relying on a single latent factor (LF)-dependent, non-negative, and multiplicative update algorithm. However, existing models' representative abilities are limited due to their specialized learning objective. To address this issue, this study proposes an α- β -divergence-generalized model that enjoys fast convergence. Its ideas are three-fold: 1) generalizing its learning objective with α- β -divergence to achieve highly accurate representation of HiDS data; 2) incorporating a generalized momentum method into parameter learning for fast convergence; and 3) implementing self-adaptation of controllable hyperparameters for excellent practicability. Empirical studies on six HiDS matrices from real RSs demonstrate that compared with state-of-the-art LF models, the proposed one achieves significant accuracy and efficiency gain to estimate huge missing data in an HiDS matrix.

How to accurately represent a high-dimensional and sparse (HiDS) user-item rating matrix is a crucial issue in implementing a recommender system. A latent factor (LF) model is one of the most popular and successful approaches to address this issue. It is developed by minimizing the errors between the observed entries and the estimated ones on an HiDS matrix. Current studies commonly employ L 2 -norm to minimize the errors because it has a smooth gradient, making a resultant LF model can accurately represent an HiDS matrix. As is well known, however, L 2 -norm is very sensitive to the outlier data or called unreliable ratings in the context of the recommender system. Unfortunately, the unreliable ratings often exist in an HiDS matrix due to some malicious users. To address this issue, this paper proposes a Smooth L 1 -norm-oriented Latent Factor (SL 1 -LF) model. Its main idea is to employ smooth L 1 -norm rather than L 2 -norm to minimize the errors, making it have both high robustness and accuracy in representing an HiDS matrix. Experimental results on four HiDS matrices generated by industrial recommender systems demonstrate that the proposed SL 1 -LF model is robust to the outlier data and has significantly higher prediction accuracy than state-of-the-art models for the missing data of an HiDS matrix.

Latent factor (LF) models are greatly efficient in extracting valuable knowledge from High-Dimensional and Sparse (HiDS) matrices which are commonly seen in many industrial applications. Stochastic gradient descent (SGD) is an efficient scheme to build an LF model, yet its convergence rate depends vastly on the learning rate which should be tuned with care. Therefore, automatic selection of an optimal learning rate for an SGD-based LF model is a significant issue. To address it, this study incorporates the principle of particle swarm optimization (PSO) into an SGD-based LF model for searching an optimal learning rate automatically. With it, we further propose an adaptive Latent Factor (ALF) model. Empirical studies on two HiDS matrices from industrial applications indicate that an ALF model obviously outperforms an LF model in terms of convergence rate, and maintains competitive prediction accuracy for missing data.

High-dimensional and sparse (HiDS) data with non-negativity constraints are commonly seen in industrial applications, such as recommender systems. They can be modeled into an HiDS matrix, from which non-negative latent factor analysis (NLFA) is highly effective in extracting useful features. Preforming NLFA on an HiDS matrix is ill-posed, desiring an effective regularization scheme for avoiding overfitting. Current models mostly adopt a standard L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> scheme, which does not consider the imbalanced distribution of known data in an HiDS matrix. From this point of view, this paper proposes an instance-frequency-weighted regularization (IR) scheme for NLFA on HiDS data. It specifies the regularization effects on each latent factors with its relevant instance count, i.e., instance-frequency, which clearly describes the known data distribution of an HiDS matrix. By doing so, it achieves finely grained modeling of regularization effects. The experimental results on HiDS matrices from industrial applications demonstrate that compared with an L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> scheme, an IR scheme enables a resultant model to achieve higher accuracy in missing data estimation of an HiDS matrix.

An adaptive latent factor model via particle swarm optimization

The valuable knowledge contained in High-dimensional and Sparse (HiDS) matrices can be efficiently extracted by a latent factor (LF) model. Regularization techniques are widely incorporated into an LF model to avoid overfitting. The regularization coefficient is very crucial to the prediction accuracy of models. However, its tuning process is time-consuming and boring. This study aims at making the regularization coefficient of a regularized LF model self-adaptive. To do so, an adaptive particle swarm optimization (APSO) algorithm is introduced into a regularized LF model to automatically select the optimal regularization coefficient. Then, to enhance the global search capability of particles, we further propose an APSO and particle swarm optimization (PSO)-incorporated (AP) algorithm, thereby achieving an AP-based LF (APLF) model. Experimental results on four HiDS matrices generated by real applications demonstrate that an APLF model can achieve an automatic selection of regularization coefficient, and is superior to a regularized LF model in terms of prediction accuracy and computational efficiency.

High-dimensional and sparse (HiDS) matrices are commonly encountered in many big-data-related and industrial applications like recommender systems. When acquiring useful patterns from them, nonnegative matrix factorization (NMF) models have proven to be highly effective owing to their fine representativeness of the nonnegative data. However, current NMF techniques suffer from: 1) inefficiency in addressing HiDS matrices; and 2) constraints in their training schemes. To address these issues, this paper proposes to extract nonnegative latent factors (NLFs) from HiDS matrices via a novel inherently NLF (INLF) model. It bridges the output factors and decision variables via a single-element-dependent mapping function, thereby making the parameter training unconstrained and compatible with general training schemes on the premise of maintaining the nonnegativity constraints. Experimental results on six HiDS matrices arising from industrial applications indicate that INLF is able to acquire NLFs from them more efficiently than any existing method does.