Movie recommendation model based on probabilistic matrix decomposition using hybrid AdaBoost integration

Probabilistic matrix factorization (PMF)

PMF was put forward by Mnih & Salakhutdinov (2007). It is a famous approach for recommendation systems. Table 1 describes the notations of PMF; Fig. 1 depicts the graphical model of PMF. It is supposed that M users, N items, and a rating matrix R ∈ R^k×N and item latent matrix R ∈ R^k×M are used to reconstruct the rating matrix R. The PMF goal is to identify the optimal matrix U,V. and minimize the loss function ɛ, which is shown below: (refer to Table 2 for parameter description) (1)${\displaystyle min\varepsilon \left(U,V\right)=\sum _i^N\sum _j^M\frac{I_{ij}}{2}{\left(r_{ij}-u_i^T{v}_j\right)}^2+\frac{{\lambda }_U}{2}\sum _i^N{∥u_i∥}^2+\frac{{\lambda }_V}{2}\sum _i^M{∥u_j∥}^2}$

After determining the objective function, the model iteratively updates U and V using the stochastic gradient descent method to minimize the function.

(2)${\displaystyle u_i\leftarrow u_i-\alpha \cdot \left(\left(r_{ij}-u_i^T{v}_j\right)v_j+\lambda u_i\right)}$ (3)${\displaystyle v_j\leftarrow v_j-\alpha \cdot \left(\left(r_{ij}-u_i^T{v}_j\right)u_i+\lambda v_j\right)}$

In the function, α is the learning rate. The iteration stops when a certain number of iterations is met or the objective function change is less than a certain threshold. Finally, the score is predicted by the trained U, V feature matrix.

Fuzzy c-means (FCM)

Fuzzy c-means (FCM) is an unsupervised clustering algorithm. Each point has a certain strength of association between the nodes and the particular community (Katarya & Verma, 2018).

The objective function J_f of FCM is below: (4)${\displaystyle J_f\left(\overrightarrow{U},\overrightarrow{C}\right)=\sum _{i=1}^n\sum _{j=1}^k{u}_{if}^f{∥x_i-c_j∥}^2}$ where u_ij is the membership degree of the i-th node to the j-th cluster and $d_{ij}=∥x_i-c_j∥$ is the distanced between the i-th node and the center of the j-th cluster. During optimizing J_f, the constraint ${\sum }_{j=1}^k{u}_{ij}=1$ must be reached. The membership function is a concept of fuzzy logic, which represents the degree of belonging to a certain category. Generally, the value ranges from 0 to 1. The 0 means not belonging to a certain category, the 1 means belonging to a certain category, and the rest of the intermediate values represent the true degree of belonging to a certain category. The objective function means being as close as possible to the class center of the same class and as far as possible from the class center of a different class. The research aim is mainly to get the minimum value. As f turns out to be larger, the process is fuzzier. The c_j can be worked out by the equation below (Salakhutdinov & Mnih, 2008): (5)${\displaystyle c_j=\frac{\sum _{i=1}^n{u}_{ij}^f{x}_i}{\sum _{i=1}^n{u}_{ij}^f}}$

The u_ij can be worked out by the equation below: (6)${\displaystyle u_{ij}=\frac{1}{\sum _{l=1}^k{\left(d_{ij}/d_{lj}\right)}^{2/\left(f-1\right)}}}$

The J_f can be minimized by iterative optimization with the update of membership degree u_ij and the cluster center c_j (Katarya & Verma, 2018).

Ensemble learning

Ensemble learning means using a series of base learners for learning, and then based on a certain rule to integrate the learning results to obtain a better learning method than a single learner. Usually there are distinctions between base learners, including differing algorithms or the same algorithm with differing parameters or hyperparameters. In general, the greater the distinction between base learners, the better the final learning outcome. Ensemble learning has an obvious edge in performance improvement. That is why it is extensively used in theoretical research and practical applications. The significant ensemble learning methods are mainly Bagging and Boosting. The AdaBoost method is used in this study, so we would introduce this method’s principle in detail (Zhang et al., 2021).

AdaBoost (Ichihashi et al., 2008) is one of the most successful ensemble learning algorithms that iteratively selects several classifier instances by maintaining an adaptive weight distribution over the training examples. AdaBoost forms a linear combination 20 of selected classifier instances to create an overall ensemble. AdaBoost-based ensembles rarely over-fit a solution even if a large number of base classifiers in-stance are used (Lei & He, 2017) and it minimizes an exponential loss function by fitting a stage-wise additive model (Wu & Wu, 2013). As the minimization of classification error implies an optimization of a non-smooth, non-differentiable cost function which 25 can be best approximated by an exponential loss (Koren, Bell & Volinsky, 2009), AdaBoost therefore per-forms extremely well over a wide range of classification problems.

AdaBoost’s algorithm idea is to combine the outputs of multiple weak classifiers to produce a more efficient classification. The main steps of the algorithm are to select the weak classifier and sample dataset, select m groups of data from the dataset as the training set, and the training weight of the dataset is 1/m. Then the weak classifier is used to iterate the training T times. After each training, the training data is updated according to the training output, and a larger weight is given to the training data that failed to classify, which gives more attention to the training failure data during the next weak classifier training. A sequence of classifier functions f₁, f₂, f₃, …, f_T is obtained by repeated iteration training of the weak classifier. At the same time, each classifier is given a corresponding weight, and the function with the better classification result has the greater weight. After T iterations of training, the final strong classifier is weighted by the weak classifier (Bao-an, Haijing & Jin-xiang, 2001).

Figure 2 indicates the structure of AdaBoost Model.

Artificial neural network

Neural networks are composed of three parts: input layer, hidden layer and output layer, which can achieve continuous nonlinear mapping.

Neural network is a multilayer feed-forward neural network characterized by forward propagation of signals and backpropagation of errors. In the forward transmission process, the signal is processed layer by layer from the input layer, through the hidden layer, and then to the output layer. Figure 3 depicts the typical architecture of a three layer artificial neural network model (Wang, Wang & Yeung, 2015).

Each connection is associated with a numeric number called weight. The output, h_i, of neuron i in the hidden layer is (7)${\displaystyle h_i=\sigma \left(\sum _{j=1}^N{W}_{ij}{x}_j+T_i^{hid}\right).}$ where σ() is called activation (or transfer) function, N the number of input neurons, W_ij the weights, W_ij inputs to the input neurons, and $T_i^{hid}$ the threshold terms of the hidden neurons. The purpose of the activation function is, besides introducing nonlinearity into the neural network, to bound the value of the neuron so that divergent neurons do not paralyze the neural network. A common instance of the activation function is the sigmoid (or logistic) function defined as: (8)${\displaystyle \sigma \left(u\right)=\frac{1}{1+exp\left(-u\right)}}$

BP neural network is a multi-layer feed-forward neural network featured by forward propagation of signals and backpropagation of errors. In the forward transmission process, the signal is processed from the input layer through the hidden layer, and finally to the output layer.

Figure 3 indicates the basic processing framework for BP neural networks, where Z = (Z₁, Z₂, …, Z_n) is a set of n values from an external input or output from another neuron; W = (W₁, W₂, …, W_n) is the weight, illustrating the strength of the connection between a neuron and other neurons; ∑WZ is an activation value, which is equivalent to the total input of an artificial neuron; O means the output of a neuron; b is the threshold of that neuron. The weighted sum of the signs is higher than b, and the artificial neuron is activated. Accordingly, the output of artificial neurons can be depicted here: (9)${\displaystyle O=f\left(\sum WZ-b\right)}$

In Eq. (7), f(⋅) is called an activation function. This activation function in this study is a nonlinear transformation function, which is the bipolar sigmoid -shaped function (tanh(x) function). In the procedure of error backpropagation, there exists the problem of differentiation of the activation function. The tanh(x) function can effectively address the point of derivative discontinuity and the matter of zero-centered output, that is why it is the activation function in the study. It is defined as follows: (10)${\displaystyle f\left(x\right)=\left(1-e^{-x}\right)/\left(1+e^x\right)}$

This article uses a three-layer BP neural network with one hidden layer structure to simulate the change of the model.

Application of probability matrix factorization model

In order to further improve the accuracy of PMF model in predicting score, the FCM method is used for score data in preprocessing data. Firstly, the FCM algorithm is suitable for solving the problems of high-dimensional and sparse data and has the advantages of strong scalability; secondly, it can solve the shortcomings of hard clustering, but express the degree of a class’s score belonging to a class in the form of membership function. Finally, this method enhances the robustness of data (Luo et al., 2014).

Algorithm thought

Given the user’s score R, FCM uses membership degree u_r,j to represent the degree of association between user u and cluster j. Assuming that there are n user scores and k clusters, the results of fuzzy clustering satisfy the following three conditions at the same time: (1) for each user’s score r and cluster j, there exists 0 ≤ u_r,j ≤ 1. The degree to truth which the score r is attributed to the clustering center j. (2) for each user’s score r, there is ${\sum }_{\mathrm{j}=1}^{\mathrm{k}}{u}_{\mathrm{r},\mathrm{j}}=1$. The score r belongs to the sum of all the different categories should be 1. (3) for each cluster j, there is $0<{\sum }_{\mathrm{j}=1}^{\mathrm{n}}{u}_{\mathrm{r},\mathrm{j}}<\mathrm{n}$. For each cluster, the sum of all membership degrees in the cluster does not exceed the total number of users.

In fuzzy clustering algorithm, a membership matrix needs to be generated, and a fuzzy similarity matrix needs to be constructed for data similarity in matrix. The methods of constructing fuzzy similarity matrix include maximum and minimum calculation method, cosine angle method, and correlation coefficient method. In terms of parameter selection, the correlation coefficient method can get better clustering effect, which is generally the default method. This article mainly adopts the correlation coefficient method.

Algorithm description

In Fig. 4, we show the workflow of the model. Firstly, the rating matrix is constructed from the training data, and then FCM is used to calculate the similarity and complete the clustering. Finally, the clustered data is applied to the PMF model to predict the user’s rating matrix.

PMF model based on AdaBoost ensemble method

The efficiency and prediction accuracy of the PMF model and the recommendation algorithm based on similarity have been greatly improved. However, due to the randomness and high-dimensional sparsity of data, the model is unstable, which affects the accuracy of recommendation.

The AdaBoost ensemble learning method solves the problem of low accuracy of a single weak learning algorithm and has good generalization ability, which effectively improves the accuracy of rating prediction.

Algorithm introduction

For the regression task, (x, y) represents a piece of data on dataset D, where x is the eigenvector and Y is the real value. We train with the multiple regression model, put the features into the regression model and produce the corresponding prediction value Φ(x, D). The predicted values on multiple data sets are integrated according to the strategy D. (11)${\displaystyle {\Phi }_A\left(x\right)=E_D\Phi \left(x,D\right)}$

where x represents the input data and Y is the output value, then (12)${\displaystyle E_D{\left(y-\Phi \left(x,D\right)\right)}^2=y^2-2y{E}_D\Phi \left(x,D\right)+E_D{\Phi }^2\left(x,D\right)}$

Eq. (9) and inequality $E{Z}^2\ge {\left(EZ\right)}^2$, then Eq. (10) can be changed into the following: (13)${\displaystyle E_D{\left(y-\Phi \left(x,D\right)\right)}^2\ge {\left(y-{\Phi }_A\left(x\right)\right)}^2}$

It can be seen from inequality (Eq. (11)) that the root mean square error of the predicted value ${\Phi }_A\left(x\right)$ generated by ensemble is smaller than the average value of root mean square error Φ(x, D), and the more unstable $\Phi \left(x,D\right)$ is, the greater the performance improvement of the ensemble method of the model.

Algorithm description

Figure 5 shows the overall structure of the model and, and the algorithm is described as follows

Relevant parameter settings

In this experiment, we take 80% of the data as training data and 20% as test data. The relevant parameters in the experiment are set to λ_U = λ_V = λ_bu = λ_bi = 0.01, the learning rate of SGD is α = 0.03. The number of hidden layers of BP is 100. The experimental data sets are Movielens and FilmTrust, which are respectively applied to PMF, FCM-PMF, AdaBoost-SVM-PMF and AdaBoost-BP-PMF models for comparison and conclusion.

This experiment is carried out on Movielens and FilmTrust datasets, both of which contain the user’s rating information of the project. Both of the data sets are high-dimensional sparse matrices, with score values of 1-5 discrete values, and sparsity of 4.47% and 1.04% respectively. Table 3 as below shows the specific information of two data sets:

Experiment

To verify the actual recommendation accuracy effect of the model, the PMF model based on hybrid AdaBoost ensemble method is evaluated by experiments, and the results are compared with the models PMF, FCM-PMF, Bagging-BP-PMF, AdaBoost-SVM-PMF in two datasets. Table 4 shows that the comparison results of the different models in the two datasets as below:

It can be seen from Table 3, we can conclude that the performance of AdaBoost method in these models is better than the bagging method. The RMSE and MAE of the FCM-Bagging-BP-PMF model and the PMF model are about 0.79765 and 0.73704, respectively. The RMSE and MAE of the AdaBoost-BP-PMF model and the AdaBoost-BP-PMF model are about 0.77215 and 0.71836, respectively, which were improved by 2.55% and 1.87%. Compare with the performance of RMSE and MAE, AdaBoost- SVM-PMF is only better than the PMF model, lower than the other models. Finally, compared with the FCM-PMF model, the RMSE and MAE of the PMF model based on the hybrid AdaBoost method were improved by 6.495% and 2.295%. The results of the five models in RMSE and MAE are shown in Figs. 6 and 7.

It can be seen from Table 5 that the overall performance of the five models. The RMSE and MAE of the AdaBoost method is better than the other models’ method and they are about 1.37855 and 1.79514, respectively. The RMSE and MAE of FCM-Bagging-BP-PMF model method are about 1.39723 and 1.79593, respectively. Compared with the FCM-Bagging-BP-PMF method, the MAE is similar, but RMSE is improved by 1.1.87%. Finally, compared with the FCM-PMF model, the RMSE and MAE of the PMF model based on hybrid AdaBoost method were improved by 2.56% and 2.8%. The comparison results of the five models in RMSE and MAE are shown in Figs. 8 and 9.