An Embedding Based Factorization Machine Approach for Web Service QoS Prediction

Abstract

As an important property of Web services, Quality of Service (QoS) is usually engaged for describing the non-functional characteristics of Web services. However, QoS value is considerable sparse since users only invoke a limited number of services in the real-world applications. In this way, predicting QoS value is a good choice to solve such ‘sparsity’ problem. Although several methods have been proposed to predict QoS value for users, most of them are always time-consuming and expensive to implement.

To solve the drawbacks of high dimensionality and huge sparse, we introduce embedding technique to map data from resource space to target space in injective and structural-preserving way. To efficiently express pairwise interactions in sparse datasets, we further introduce factorization machine, which is an impactful algorithm to deal with sparse data prediction in the world of machine learning and can be computed in linear time.

Based on the above characteristics of our scenario and the advantages of factorization machine and embedding, this paper proposes an embedding based factorization machine approach to predict missing QoS values for Web services. First of all, user id and service id are encoded by one-hot encoding. And then, the one-hot encoding of user id and service id are mapped to different embedding vectors. Finally, the embedding vectors are regarded as implicit vectors and the idea of factorization machine is exploited to make missing QoS value prediction. Experiments on real-world dataset validate the effectiveness of our approach, which outperforms the other state-of-the-art methods in terms of QoS prediction accuracy.

1 Introduction

Web services are self-described software applications designed to support interoperable machine-to-machine interaction over a network via standard interfaces and communication protocols [17]. With the increasing adoption of Service Oriented Architecture (SOA), the number of Web services are increasing rapidly on the Internet. Among the massive Web services, lots of them provide similar functions. However, the quality of these similar Web services is different. Quality of Service (QoS) is usually used to describe the nonfunctional characteristics of Web services [16]. In general, the QoS of Web services has some properties, including price, response time, throughput, reliability, availability, etc. In the scenario of service selection, QoS is an important factor to impact users’ decision. How to choose the services that satisfy users is an urgent problem. On the basis of functional matching, considering the QoS of Web services is an effective solution. Therefore, investigating Web services QoS is becoming more and more important.

Previous QoS-based studies have been applied for Web service recommendation, Web service composition, Web service selection, and so on. It’s essential to provide accurate QoS value of Web services to make these QoS-based approaches work well. In general, the QoS value of Web services can be measured on the server-side or client-side. As for measured by server-side (e.g., price, popularity, etc.), QoS value is usually declared by service providers or the third party (e.g., UDDI), and it is identical for different users. The QoS value measured by client-side (e.g., response-time, throughput, availability, etc.) is closely related to network status, geographical location, service runtime environment, etc. Therefore, the QoS experience of different users is quite different. In reality, a service user usually only invokes a limited number of Web services in the past and thus only observes QoS values of these invoked Web services. For the sake of obtaining accurate and personalized QoS value of users, Web service QoS evaluation is indispensable for enhancing the service users’ experience.

However, in practice, invoking Web service for evaluation purpose at the client-side is quite difficult and has the following critical drawbacks:

• 1. Web service invocations may be charged because the Web services are usually provided and hosted by other organizations. Even if the Web services are free, executing real-world Web service invocations for evaluation purposes consumes resources of service providers and imposes costs of service users.

• 2. It is time-consuming and impractical for service users to evaluate all the Web service candidates, since there are massive Web services in the Internet.

• 3. Service users are usually not experts on Web service evaluation and the common time-to-market constraints make in-depth evaluations of the target Web services difficult.

Based on the above analysis, it is an urgent task to accurately predict the missing QoS value of the candidate services for different service users, so that it can provide support for Web service recommendation, Web service composition, Web service selection, and so on. Although several methods [9, 18, 19] have been proposed to predict QoS value for users, most of them are always time-consuming and expensive to implement. In neural network, embedding maps the data from resource space to target space injective and structural-preserving. Further, the processing of embedding can solve the drawbacks of high dimensionality and huge sparse. In the world of machine learning, factorization machine can estimate interactions even in problems with huge sparsity. It is a remarkably smart way to express pairwise interactions in sparse datasets. Moreover, it can be computed in linear time and it depends only on a linear number of parameters [5].

Due to the above characteristics of our scenario and the advantages of factorization machine and embedding, we propose an Embedding based Factorization Machine approach, called EFMPred, to Predict missing QoS values for Web services. User id and service id are firstly encoded by one-hot encoding. And then, the one-hot encoding of user id and service id are mapped to embedding vectors. Specifically, the embedding vectors are regarded as implicit vectors to model the pairwise interaction between users and services. Finally, we exploit EFMPred model to make missing QoS value prediction. Extensive experiments on real-world dataset demonstrate that our EFMPred model can improve the QoS value prediction accuracy effectively.

In summary, the main contributions of this paper are as follows:

• 1. To the best of our knowledge, this is the first work applying embedding technique to traditional factorization machine for QoS prediction.

• 2. EFMPred can capture the implicit relationship between users and services by performing user id and service id embedding. In addition, EFMPred can predict missing QoS information for every user-service pair.

• 3. Extensive experiments on real-world dataset are conducted to evaluate the effectiveness of our approach. The experimental results demonstrate that our approach outperforms the other state-of-the-art baseline methods in terms of prediction accuracy.

The remainder of this paper is organized as follows. Section 2 presents our missing QoS prediction problem and gives an overview of our framework. Section 3 introduces our EFMPred model for personalized QoS value prediction. Section 4 discusses and analyses the experimental results. Related works are presented in Sect. 5 and a conclusion of this paper is drawn in Sect. 6.

2 Preliminaries

In this section, we firstly describe the QoS property prediction problem in Sect. 2.1. Then we introduce the framework of our embedding based factorization machine model in Sect. 2.2.

2.1 Problem Description

In this paper, the problem we investigate is how to accurately predict missing QoS information for service users based on the historical QoS usage experience. Given a list of users and services, users invoke services to mark the property of QoS. The detailed description is as follows.

Let $U=\{u_1,u_2,...,u_m\}$ be the set of m users and $S=\{s_1,s_2,...,s_n\}$ be the set of n Web services. The QoS value $q_{ij}$ of service $s_j$ which is observed by user $u_i$ is presented as a triple ($i,j,q_{ij}$). As mentioned before, the QoS property of Web services includes price, response time, throughput, reliability, availability, etc. Herein, $q_{ij}$ denotes a k dimension vector representing the QoS values of $k-th$ criteria. Let $\varOmega$ be the set of all QoS values, $\varDelta$ be the set of all existing known QoS values, $\varLambda = \varOmega - \varDelta$ is the set of missing QoS values. The missing QoS values $y_{ij}$ will be predicted by the existing QoS values in $\varDelta$.

A toy example is shown in Fig. 1 to better comprehend the idea of unknown QoS prediction. The interaction between users and services is illustrated by a user-service matrix as shown in Fig. 1(a), each element in the user-service matrix denotes a QoS value observed by a user on a certain service. The problem we study in this paper is then transferred to how to precisely predict the missing entries in the user-service matrix based on the existing entries. We can provide users with personalized QoS information once the missing entries are accurately predicted. We observe that although a part of the entries are already known in Fig. 1(a), every pair of users still have very few commonly invoked services. Based on the idea of collaborative filtering and our embedding based factorization machine model, we can complete the missing entries in the matrix for service users as shown in Fig. 1(b).

Fig. 1.

A toy example for missing QoS prediction

Fig. 2.

Framework of EFMPred

2.2 Prediction Framework of EFMPred

In this section, we present our missing QoS value prediction framework of EFMPred. As shown in Fig. 2, our QoS prediction framework mainly includes four components: Input Feature Vector, Embedding Vectors, EFMPred Model and Prediction Score. The detailed functionality of each component is as follows:

• 1. Input Feature Vector. Based on the historical Web service QoS data that a service user observes on a certain service, the user id and service id are encoded with one-hot encoding respectively.

• 2. Embedding Vectors. After being encoded, the one-hot encoding of user id and service id are mapped into different embedding vectors via back propagation algorithm [11].

• 3. EFMPred Model. The embedding vector of user is multiplied by the embedding vector of service (e.g. the dot product operation in factorization machine) to model the interactions between users and services.

• 4. Prediction Score. After obtaining the interactions between users and services, we can predict the missing QoS value for every pair of user-service.

Unlike the traditional factorization machine, our EFMPred model apply embedding technique to represent feature vectors. The detailed description about how to map the user id and service id to embedding vectors is shown in Fig. 3. We present the user id and service id (e.g., ID = 3) to the corresponding one-hot encoding (e.g., ‘001000’), and then, the one-hot encoding of user id and service id are mapped to the corresponding embedding vector (e.g., $V_3$) via a fully connected mapping. As can be seen in Fig. 3, the weight of red line directly corresponds to the output node value (e.g., $v_{31}$, $v_{32}$, $v_{33}$). That is the value of each dimension of the embedding vector.

Fig. 3.

Toy example of id embedding

3 Approach

In this section, we introduce our embedding based factorization machine approach for the missing QoS value prediction. Our EFMPred model mainly include two phases, we will describe them in detail in the following subsections.

3.1 Extracting Embedding Features from User ID and Service ID

The method of one-hot encoding is using n bit status register to encode the n states, each state has an independent register bit. For example, if we want to use one-hot coding to express four seasons in a whole year, we need four binary features, each of them representing one season. That is, ‘1000’ indicates spring, ‘0100’ indicates summer, ‘0010’ indicates autumn, ‘0001’ indicates winter.

In order to obtain the embedding vectors, we firstly exploit one-hot encoding to represent user id and service id. And then, we regard the one-hot encoding vectors as input of the full connection layer. Next, we apply back propagation algorithm [11] to calculate the weight of every edge in the full connection layer. The output layer of the full connection layer is embedding layer (e.g., embedding vectors of user id and service id).

Embedding maps the data from resource space to target space injective and structural-preserving. The processing of embedding can solve the drawbacks of one-hot encoding, e.g. high dimensionality and huge sparse.

3.2 Embedding Based Factorization Machine Model

After obtaining the embedding vectors of user id and service id, we exploit the idea of factorization machine to predict the missing QoS value for Web services. Given a $m*n$ user-service matrix, the factorization machine model equation is defined as:

$$\begin{aligned} \hat{y}( X ) =w_0+\sum _{i=1}^f{w_ix_i}+\sum _{i=1}^f{\sum _{j=i+1}^f{{<}V_i,V_j{>}}}x_ix_j, \end{aligned}$$

(1)

Wherein, $X=(x_1, x_2, ..., x_{m+n})$ is the feature vector, which is concatenated by user id and service id. f is the length of the feature vector. $V_i$ describes the $i-th$ embedding vector. ${<}V_i,V_j{>}$ is the dot product of the embedding vector of user id and service id. $w_0$ is the global bias, $w_i$ is the weight of the $i-th$ variable. $W=\{w_1, ..., w_f\}$, $V=\{V_1, ..., V_f\}$. A row $V_i$ within V describes the $i-th$ variable with k factors.

$$\begin{aligned} {<} V_i,V_j{>} =\sum _{l=1}^k{v_{i,l}}\cdot v_{j,l}, \end{aligned}$$

(2)

Herein, k is a hyperparameter that decides the dimensionality of the embedding vectors. $v_{i,l}$ and $v_{j,l}$ are the $l-th$ value in the $i-th$ embedding vector of user id and the $j-th$ embedding vector of service id respectively. ${<}V_i,V_j{>}$ models the interaction between users and services.

The pairwise interactions between users and services can be reformulated as follows, the detailed derivation process can be found in [5]:

$$\begin{aligned} \begin{aligned} \sum _{i=1}^f{\sum _{j=i+1}^f{{<}V_i,V_j{>}}}x_ix_j=\frac{1}{2}\sum _{l=1}^k{( ( \sum _{i=1}^f{v_{il}x_i} ) ^2-\sum _{i=1}^n{v_{il}^{2}x_{i}^{2}})}, \end{aligned} \end{aligned}$$

(3)

In order to estimate the quality of EFMPred model, a loss function should be conducted for evaluating the error between the estimated value and the original value. The square of the errors between the estimated value and the original value is usually applied to define the loss function. Therefore, the loss function of our EFMPred model is as follows:

$$\begin{aligned} \underset{\theta }{\min }L\left( y,\hat{y} \right) =\frac{1}{2}\sum _{i=1}^m{\sum _{j=1}^n{I_{ij}}}\left( y_{ij}-\hat{y}_{ij} \right) ^2, \end{aligned}$$

(4)

where, $\theta =\{w_0, W, V\}$ is the model parameters, $I_{ij}$ is the indicator function that is equal to 1 if user $u_i$ invoked Web service $v_j$ and is equal to 0 otherwise. The optimization problem of the loss function is to minimize the sum-of-squared-errors objective function.

A local minimum of the objective function given by (4) can be solved by performing stochastic gradient descent in $\theta$:

$$\begin{aligned} \frac{\partial L\left( y,\hat{y} \right) }{\partial \theta }=\left( \hat{y}-y \right) \frac{\partial \hat{y}}{\partial \theta }, \end{aligned}$$

(5)

$$\begin{aligned} \frac{\partial \hat{y}}{\partial \theta }=\left\{ \begin{array}{lc} 1, &{} if\ \ \theta =w_0\\ x_i, &{} if\ \ \theta =w_i \\ xi\sum \nolimits _{j=1}^f{v_{jl}x_j-v_{il}x_{i}^{2}}, &{} if\ \ \theta =v_{il} \\ \end{array} \right. \end{aligned}$$

(6)

3.3 Complexity Analysis

The main computation of our embedding based factorization machine model is to evaluate the object function L and its gradients against the variables. Since $x_i\sum _{j=1}^f{v_{jl}x_j-v_{il}x_{i}^{2}}$ is extensively related to l, in the process of parameters iteration, we only need to compute all of the l in the formulation $\sum _{j=1}^f{u_{jl}x_j}$ in the first time. And then, it is convenient to obtain all of the gradient of $v_{jl}$. Obviously, the complexity of computing all l in the formulation $\sum _{j=1}^f{u_{jl}x_j}$ is O(kf). When $\sum _{j=1}^f{u_{jl}x_j}$ is known, the complexity to compute every gradient of parameter is O(f). After obtaining the gradient of parameters, the complexity of updating parameters is O(1). In the EFMPred model, the number of parameters is $kf+f+1$. Therefore, the computation complexity of our EFMPred model is O(kf). In summary, our EFMPred algorithm can complete the training model in linear time. This complexity analysis demonstrates that our EFMPred model is very efficient and can be scaled to large datasets.

4 Experiment

In this section, we conduct a series of extensive experiments on a real-world dataset to validate our EFMPred model by comparing our approach with several state-of-the-art approaches and analyzing the experimental results.

In the following subsections, we describe the statistics of our dataset in Sect. 4.1. The evaluation metrics are presented in Sect. 4.2. The performance comparison between our EFMPred model and other state-of-the-art approaches is introduced in Sect. 4.3. And the impact of parameters and the analysis of experimental results are presented in Sects. 4.4 and 4.5 respectively.

4.1 Dataset Description

We conduct experiments on a publicly accessible dataset: WSDream¹. Which includes 339 service users and 5825 Web services. Moreover, it collects 1,974,675 records that service users invoke Web services, namely, the QoS properties values are observed by service users on real-world Web services. The WSDream dataset we use mainly collects the response time and throughput values. Therefore, we can obtain two $339*5825$ user-service matrices. The entries on the two user-service matrices are response time values and throughput values respectively. The ranges of response time and throughput are in the interval [0, 20] and [0, 1000] respectively (Table 1).

Table 1. Statistics of web service QoS dataset

Although we mainly focus on two QoS properties (e.g. response time and throughput) in this paper, Our EFMPred model can be easily scaled to the prediction of other QoS properties without any modification. When conducting experiments to predict the other QoS properties, the entries on user-service matrix are just need to be set to the corresponding QoS property value that observed by a user on a certain Web service.

4.2 Metrics

In statistics, Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) are evaluation metrics used to measure how close the prediction results are to the reality outcomes [12]. In the experiment, we use MAE and RMSE to evaluate the errors between our prediction results and the real QoS properties values. MAE is given by:

$$\begin{aligned} MAE=\frac{\sum _{ij}{| \hat{r}_{ij} - r_{ij}|}}{N}, \end{aligned}$$

(7)

and RMSE is defined as follows:

$$\begin{aligned} RMSE=\sqrt{\frac{\sum _{ij}{( \hat{r}_{ij}-r_{ij}) ^2}}{N}}. \end{aligned}$$

(8)

where, N represents the number of predicted QoS values, $\hat{r}_{ij}$ represents the predicted QoS value of service $s_j$ observed by user $u_i$, while $r_{ij}$ is the real QoS value of service $s_j$ would be observed by user $u_i$ in the dataset. Smaller MAE and RMSE values indicate better performance.

4.3 Performance Comparison

In this section, in order to verify the effectiveness of our EFMPred model, we compare our approach with the following approaches:

• 1. UMEAN (user mean). This approach employs a service user’s average QoS value on the used Web services to predict the QoS values of the unused Web services.

• 2. IMEAN (item mean). This approach employs the average QoS value of the Web services observed by other service users to predict the QoS value for a service user who never invoke this Web service previously.

• 3. UPCC (user-based collaborative filtering method using PCC) [8]. This approach is a very classical approach. In this paper, it employs similar users for the QoS value prediction.

• 4. IPCC (item-based collaborative filtering method using PCC) [6]. This approach is widely used in industry company. In this paper, it employs similar Web services for the QoS value prediction.

• 5. UIPCC [18]. This approach employs both of the user-based and item-based collaborative filtering approaches to make QoS prediction. It utilizes the historical QoS usage experience from similar users and similar services to predict missing QoS values.

• 6. PMF (probabilistic matrix factorization) [7]. This approach is proposed by Salakhutdinov and Minh, and it employs user-item matrix to predict the missing QoS value.

• 7. NMF (nonnegative matrix factorization) [2]. This approach is proposed by Lee and Seung. Compared to traditional matrix factorization approach, it mainly add an extra constraint to the model. That is the values in the factorized latent factors must be nonnegative.

• 8. NIMF (neighborhood-integrated matrix factorization) [19]. This approach employs the concept of user-collaboration for Web service QoS prediction. It finds out a list of similar users for the current user and integrate the information of similar users and all available QoS values to predict missing QoS values for service users.

In practice, service users only invoke a limited number of Web services. Therefore, the user-service matrix is very sparse. In this paper, we conduct the experiments by randomly remove a part of entries on the user-service matrix to make the matrix with different sparsity (i.e., 10% to 90%). For example, 10% denotes that we remove 90% entries on the user-service matrix. And then we set the 90% entries as testing set, the remaining 10% entries as training set. The QoS prediction accuracy results are shown in Table 2. As can be seen that our EFMPred model outperforms other state-of-the-art approaches in both of the response time and throughput. The table only presents the results of matrix density that is 10% and 90%, all results and impact of matrix density will be presented and discussed in Sect. 4.4.

Table 2. QoS prediction accuracy comparison

Fig. 4.

Impact of matrix density (QoS property is response time)

4.4 Impact of Matrix Density

The matrix density is an important factor to impact QoS prediction accuracy. It represents how much QoS information observed by users on Web services we can utilize. In order to study impact of matrix density, we vary the density of user-service matrix from 10% to 90% with a step value of 10%. In addition, the embedding vector dimensionality of response time and throughput is set as 70 and 400 respectively in this experiment.

Fig. 5.

Impact of matrix density (QoS property is throughput)

Figures 4 and 5 correspondingly show the experimental results of response time and throughput. Figure 4 demonstrates that when increasing the user-service matrix density from 10% to 50%, the MAE value and RMSE value show a downward trend significantly. However, when the user-service matrix density is increased from 50% to 90%, the magnitude of decrease in MAE value and RMSE value changes to relatively slow. Figure 5 shows the same trend as Fig. 4. This observation indicates that when the matrix density is very sparse, collecting more QoS information can do great favor to QoS prediction accuracy. However, when the matrix density is come to some extent (e.g. 50%), the effect of collecting more QoS information is not quite so obvious.

Fig. 6.

Impact of embedding vector dimensionality (matrix density = 10%)

4.5 Impact of Dimensionality

Dimensionality determines how many embedding features are extracted from user id and service id, e.g. the dimensionality of embedding vectors in the EFMPred model. In order to study impact of the embedding vector dimensionality, we set the embedding vector dimensionality varying from 50 to 100 with a step value of 10 for response time. With regard to throughput, the embedding vector dimensionality is set from 100 to 1000 with a step value of 100. In this experiment, we set the matrix density as 10%. Figures 6 and 7 show the experimental results of response time and throughput respectively. Figure 6 demonstrates that when increasing the embedding vector dimensionality from 50 to 70, the MAE value and RMSE value show a downward trend. While the embedding vector dimensionality is from 70 to 100, the MAE value and RMSE value show an upward tendency. This observation shows that relatively large dimensionality (e.g., 70) can enhance the QoS prediction accuracy. However, if the dimensionality is set too large will cause the overfitting problem, which will potentially hurt the prediction quality. Figure 7 shows the same trend as Fig. 6. But we can obtain the lowest MAE value and RMSE value under the condition of the embedding vector dimensionality is equal to 400, because the range of throughput is wider than response time.

Fig. 7.

Impact of embedding vector dimensionality (matrix density = 10%)

5 Related Work

In this section, we present some related investigations on missing QoS value prediction. In the field of service oriented computing, most of the QoS prediction approaches are based on collaborative filtering. The collaborative filtering approaches can be categorized into three types: memory-based, model-based and hybrid. In the following, we will discuss them separately.

The memory-based collaborative filtering approach employs users’ historical records to compute the similarity between users or items. This approach includes user-based approach, item-based approach, and the hybrid of them. User-based approaches predict the missing QoS values by measuring the similarity between users. Item-based approaches predict the missing QoS values for the current user by measuring the similarity between services. Examples of memory-based approaches are as follows. Sun et al. [10] present a new similarity measure for Web service similarity computation and propose a novel normal recovery collaborative filtering approach. Xiong et al. [14] propose a collaborative approach to QoS prediction of Web services on unbalanced data distribution. Ma et al. [4] propose an unknown QoS values prediction algorithm to realize some important characteristics of objective QoS datasets.

The model-based collaborative filtering approach tend to apply machine learning algorithms to predict users’ rating of unrated items. Examples of model-based approaches are as follows. Wu et al. [13] propose a novel two-phase K-means clustering based credibility-aware QoS prediction method to address the problem of unreliable data offered by untrustworthy users. Yu et al. [15] propose a novel neighbor factor model by taking the latent physical location and network status information into consideration. Zheng et al. [19] propose a neighborhood integrated matrix factorization approach for Web services QoS prediction by taking advantages of the past Web service usage experiences of service users.

The hybrid collaborative filtering approach integrates the memory-based and the model-based collaborative filtering approaches. Examples of hybrid approaches are as follows. Chen et al. [1] propose a neighborhood regularized matrix factorization method by properly incorporating both user and Web service neighborhood relationships. Su et al. [9] present a novel hybrid Web service QoS prediction approach by integrating the direct similarity and transitive indirect similarity of services. Lo et al. [3] propose an extended matrix factorization framework with relational regularization terms inside a neighborhood.

The above memory-based, model-based and hybrid methods can solve the problem of missing QoS prediction to some extent. However, the drawbacks of these approaches are huge sparsity, high time complexity and expensive to implement. But our EFMPred model employs the embedding features extracting techniques to represent the vectors of user id and service id. The embedding vectors can model the interactions between users and Web services. In addition, our embedding based factorization machine model can solve the problem of data sparsity and can be calculated in linear time.

6 Conclusion

This paper proposes an embedding based factorization machine approach to predict missing QoS values for Web service in order to improve the prediction accuracy and avoid time-consuming and expensive Web services invocation. By employing embedding technique and factorization machine model, we can reduce the dimensionality of vectors, learn features with large data sparsity and our EFMPred can be computed in linear time. Therefore, our EFMPred model can be easily extended to large data sets. Comprehensive experiments demonstrate that our approach outperforms other state-of-the-art baseline approaches.

At present, we mainly focus on response time and throughput value prediction. In future, we’d like to apply our model to other QoS properties of the real-world Web services (i.e., failure tolerance, reliability, availability, etc.). Other characteristics of service users and Web services can also be taken into consideration to integrate them into our model.