Keywords

1 Introduction

The number of mobile technology users has been increasing on a large scale [1]. The features of smartphones have been expanded to meet the different needs of users [2]. Mobile applications are able to offer a vast and distinct set of pleasurable experiences, that often arouse in users the desire to engage for a long time in tasks that generate value, fulfillment and well-being [3]. In today’s society, the mobile applications’ market is fast growing, causing changes in society. Nowadays, mobile apps are an integral part of everyday life in society, influencing how people interact with the world. With the advent of mobile apps, our lives have become richer, facilitating global social interactions [4].

Among several mobile applications that have been proposed, many follow a crowdsourcing approach. According to Bassi et al. [5] “crowdsourcing is a type of participatory online activity in which an individual or organization proposes to a group of individuals of varying knowledge and number, via a flexible open call, the voluntary undertaking of a task”. Ghezzi et al. [6] affirm that “crowdsourcing is a branch of co-creation practice, which has been made possible through the upsurge of the web, where the “crowd” can help in validating, modifying and improving a company’s value - creating idea or the material it posts over the internet”. When crowdsourcing is coupled with mobile technology it is usually called mobile crowdsourcing (MC). In a MC app, the crowd of users, through mobile devices, perform various tasks, such as sharing information, results, analyzing data, among others [7]. Mobile applications, made available by most popular mobile devices – smartphones and tablets – provide a number of useful features, making everyday life easier for people in general. For example, if we need location services for roads in the city, the service function in mobile applications is to provide navigation, map and GPS [8].

Jung et al. [9] conducted a product survival analysis study for the Apple App Store in Korea. They said customer ratings, ranking, and content size offer a great deal of weight for a product, including apps, to stay in the market, especially if it’s free. Therefore, the app ranking, or user rating, influences popularity as it brings more users to use these applications. In addition, for an MC application to succeed, it needs to go beyond being used, it is necessary that the users engage with it [10]. According to Lalmas et al. [11], user engagement is about the quality of the user experience, emphasizing aspects of interacting with an online application and, in particular, the desire to use that application more times. Liang et al. [12] define engagement as a psychological state where an individual invests his cognitive, physical, and affective energy to solve a task. The existence of factors that influence the users engagement in diverse applications is already explored in the literature, such as usability and confidence [11].

Some types of MC apps involve recommendation systems, benefiting from user evaluations so that they can provide rich information necessary for the app to make more recommendations to the user, in an efficient way [13]. This evaluation covers comments on the services or items recommended by the application. Recommendation systems have always faced the problem of sparse data. In the current era, however, with its demand for highly personalized, context-aware recommendations, real-time, the sparse data problem only threatens to worsen. If the app does not succeed in engaging the user, the user can not get the information needed to make the recommendations. Therefore, applications that involve crowdsourcing allied to recommendation systems have an increased need for engagement [13].

The textual part of the rating mechanism, represented by the comments section, is a free text description, without any predefined structure, and is used to describe, in a totally informal way, impressions, positions, claims, bugs, and desired resources [14]. These comments can provide valuable insight into a number of highly relevant topics, and developers can use this feature to better meet users expectations, by designing ways to engage them [15]. If app developers fail to engage users, and if the relevant suggestions for improving the app are ignored, the app rating will decrease and the app will probably lose market share [16].

In this work, our main purpose was to verify how negative comments—or complaints—, related to factors that knowingly influence engagement, influence the users’ evaluation about crowdsourcing apps. We investigated the following hypothesis: the lower the number of complaints related to engagement factors, the higher the app rating. We performed an experiment, in which we selected a set of apps from Android Play Store and analyzed the comments of users to determine the number of complaints involving factors that influence engagement. We then calculated a linear correlation to verify the influence of those factors over the users’ rating of the selected apps.

This paper is divided into six sections, including this one. In Sect. 2, we introduce a theoretical reference of all the concepts that will be used in our experiment. In Sect. 3, we present the related works in the literature. In Sect. 4, we describe our methodology and the experiment. In Sect. 5, we present the results. And finally, in Sect. 6, we present our conclusions and future work.

2 Theoretical Reference

In this study it is important to identify concepts such as: mobile applications and how they are made available to the users, the meaning of crowdsourcing and, more specifically, mobile crowdsourcing, conceptualizing recommendation systems - citing examples, and finally elucidating the definition of user - citing some factors that influence the user’s real engagement.

2.1 Mobile Applications and App Stores

With the arrival of smartphones, the way users request software services was revolutionized [17]. Consequently, smartphone functionalities have been greatly expanded to meet the different needs of users [2]. Several services were created by the opportunities generated by the increased use of mobile applications [17]. Mobile applications are one of the fastest growing segments in the software application download markets. Several mobile app markets, such as the Amazon App Store, BlackBerry World, Google Play and the Apple App Store, emerged and grew rapidly in a short period of time. These application store markets exhibit characteristics of a “long tail market”, as a huge plurality of digital products and low user search costs [18].

Currently, the Android operating system is the most used open source smartphone [19]. Android applications can be found at the Google Play Store that offers a variety of apps to download on any device that has the operating system installed. These applications can be free or paid. Hundreds of thousands of programmers submit applications for Google Play Store. Millions of people create an account on the Google Play Store to use the various apps it offers [20].

Each app in the Play Store has developer information, classifications, ratings, and space where users post comments informing their opinion of the app. Each comment has two characteristics: The first is that they are written in short sentences. The second is that the comment refers only to one version of the application and it may have been updated over time [21]. Evaluations and comments are of great importance to the success of applications [22].

Although having a large number of applications available can be an advantage, choosing the best option to meet a need can be an extremely complex task for the user. Users’ reviews provide a way for helping other users in their choices. Reviews are rich repositories of information where multiple users post a rating on a star rating and/or comment on app quality, bugs, human-computer interaction issues such as usability and more. This information helps the user in deciding whether to install or purchase an application, whether to hire a service or not, among other things [23, 24]. However, little has been researched on user review on mobile apps [25].

2.2 Crowdsourcing Applications

Crowdsourcing is a distributed model of participatory online activities where an undetermined crowd of people works engaged in solving a given task through an open call [5, 26]. Crowdsourcing is, for the most part, a very well structured process by the company or organization that proposes the task, because of this it is able to make the most of the individuals’ intelligence and creativity in a targeted way [27].

When crowdsourcing is coupled with mobile technology it is called mobile crowdsourcing (MC). In an MC app, the crowd of users, through mobile devices, perform various tasks, such as sharing information, results, analyzing data, among others [7]. One feature of Mobile Crowdsourcing applications is the active and passive contribution of the crowd. In the active contribution, users generate data such as text translation, user evaluation, performing calculations, or entering input data as a solution. In the passive contribution, the data from the features of the user’s smartphone, such as sensors, GPS, triangulation calculations of a position, among others. It should be noted that the passive contribution crowdsourcing tasks are performed transparently for the device owner [26].

MC applications are a new way for commercial crowdsourcing [28]. Mobile crowdsourcing markets have attracted great attention from the industrial and academic community [29]. Several MC applications have stood out in the use by users like: review of films (IMDBFootnote 1, NetflixFootnote 2), e-commerce (Google PlayFootnote 3, AmazonFootnote 4), provision of services (UberFootnote 5, AirbnbFootnote 6), among others.

2.3 Recommendation Systems

A Recommendation System is a system that uses a set of techniques and software tools to generate suggestions for items [30, 31]. Item is the name given to what the recommendation system suggests. Some examples are movies, a product, a news, among others. Recommendation Systems generally are directed to recommending a particular type of item. So your interface, your set of techniques and your algorithms are customized to get a more accurate recommendation. They are considered important in big companies like Amazon, YouTube, Netflix, among others [30].

They can be categorized into three forms: Content Based Recommendation System, Collaborative Filtering Recommendation System, and Hybrid Recommendation System [32]. A Content-Based Recommendation System uses user information such as profile, behavior, and choices to generate recommendations. An example can be considered an online movie rental. In this case, the system will store which movies the user has been watching for a period of time and thus suggest new movies based on what he has watched. If he has ever watched action movies, the system may suggest launching more action. The Collaborative Filtering Recommendation System uses choices that have been made previously by users who have similarities. An example might be a user who wants to buy a book at an online bookstore. When he enters the site when fetching books, he will be recommended by similar books based on the features previously searched for. And a Hybrid Recommendation System makes recommendations using the two previous forms. An example can be a user that connects to Facebook and receives recommendations from friends based on their tastes, preferences and also based on their user profile [32].

2.4 User Engagement

User engagement is an essential concept in application design. We can say that successful applications are not only used, but encourage the user to invest time, attention, and emotion, seeking to meet their needs. User engagement is not a new concept, but has been stimulating an evident number of researchers from diverse areas, such as information science, computing and learning sciences. Currently, we are in a highly connected society, engaging the user has become a non-trivial goal and an undeniable need [11]. People indulge emotionally, physically, and psychologically when engaged in performing their role [33].

Measuring user engagement is essential to assess whether applications are able to engage users effectively. User engagement is a complicated phenomenon - this gives rise to several measurement approaches. Existing literature means to assess user involvement include the use of questionnaires, observational methods, and facial expression analysis [11].

From the literature studied, it is known that some factors influence the user’s engagement in applications, they are:

  • Usability - As for the application interface, it is important to perform the tasks without any difficulty.

  • Aesthetic Appeal - As for the application interface, it is very important that it be aesthetically appealing.

  • Attention - As for the use of the application, it is important that the user is fully concentrated, without perceiving the passage of time.

  • Endurability - It is important that the application motivates the user to use it frequently and to share it with friends.

  • User Control - It is critical that the user realize that he is in control of the application until he reaches his goal.

  • Interactivity - It is critical that the application promote easy, simple and fluid interaction.

  • Pleasure - It’s important to be fully involved with the application, providing a satisfying and rewarding experience.

  • Sensory Appeal - It is important that the application uses sensory features, such as: hearing, speech, vision and touch.

  • Confidence - The application must pass credibility and trust throughout the user experience.

  • Efficiency - The recommendations suggested by the application should be fully compliant with user preferences.

Typically, recommendation systems were evaluated exclusively through the precision of the algorithms that guide recommendations. However, we have noticed concerns about the importance of identifying the factors that influence users to engage with a recommendation technology, since only a good recommendation does not guarantee an effective, efficient and satisfactory user experience.

3 Related Work

There have been many recent studies that have investigated the factors involved in users’ adoption, intent of use and acceptance of apps. However, there have not been many studies that have investigated the engagement factors that influence the users’ evaluation in mobile app stores.

Table 1. Some related works
Full size table

In [34], Harris et al. investigated the factors that influenced the user to install a mobile application - a model is created, using perceived risk, perceived benefit, confidence and intent to install. Seven antecedents of trust and risk including perceived safety, perceived reputation, application characteristics, familiarity, desensitization, consumer willingness to trust and consumer disposition. The presented model explains 50.5% of the variation in the intention to install an application The results show that the consumers who perceive more security have greater confidence and less perceived risk.

In [35], Kang et al. presented as the performance enhancement of tasks, the ease, the opinions of others important, the motivation of the entertainment, the information seek motivation and motivation of social connection could predict the intention to use mobile applications by users. The results of a hypothetical model test show that ease is the key factor influencing the intention of continuing the use of mobile applications.

In [36], Yang et al. integrated the Technology Acceptance Model, the Planned Behavior Theory and the Usage and Gratification Theory to predict the intent and use of mobile applications of young American users. The model was tested by a Web search of 555 American college students. The perceived pleasure, ease of use, utility and subjective norm emerge as significant predictors of their attitudes in mobile applications. The results conclude that the attitudes and intentions of such young people predict the use of mobile applications.

In [37], Wu et al. conducted an online survey to explore the factors that influence consumers’ intent to continuously use branded applications. The results confirm the great importance of application engagement, which is positively influenced by the expectation of effort, social influence and brand identification. Moreover, the interactivity perceived by consumers positively affects the expectation of effort, which, in turn, contributes to the expectation of performance. The expectation of performance is another direct factor of the intention to use continuity. The paper suggests that marketers of branded apps need to emphasize improving consumers’ app engagement rather than just providing useful app functions.

In summary, there were no studies found that investigated the engagement factors that influence the users’ evaluation in mobile app stores, as we can see in Table 1, but a few studies were found that investigated the factors that influence adoption and the continuous use of apps.

Fig. 1.
figure 1

Each step of our methodology.

Full size image

4 Methodology

Our methodology was divided into four phases: (i) select apps given a specific category, (ii) choose applications that have already been selected and have valid N comments, (iii) interpret the comments according to the engagement factors, and (iv) identify how the engagement factors induce the evaluation of the users. In Fig. 1 we present each step.

In the execution of this methodology, for data collection, we first select apps about service recommendations from the Android Play Store - chose the android operating system because it is the most used open source in mobile devices [19]. This category - service recommendations - was chosen because it is not well explored in the literature, and has as a principle the recommendation that uses the user’s evaluation to make the recommendation more efficient. The applications selected were those that hold the highest amount of comments according to category: recommendation of services. The following applications have been selected:

  • Service Touch: Application for users to hire or offer services without intermediaries, such as: painter, day laborer, hairdresser, mechanic, personal trainer, electrician, masseur among others.

  • Labor: Application to search for service providers.

  • Get Ninjas: Application for professionals to offer their services and for clients to hire them.

  • Helpie: Application to search for service providers.

  • Service Market: Application to search for service providers.

  • DiaríssimaFootnote 7: Application to search for domestic service providers.

  • Help me cleaning: Application to search for domestic service providers.

Table 2. Examples and responses quantities for each factor
Full size table

After this selection, we choose the 10 most recent/really useful comments for each chosen application. 10 comments were chosen because of the limited number of experts on the team in this work. We interpreted each comment manually, in order to identify whether or not some engagement term can be applied to the due complaint. We pointed out all the complaints for each engagement factor - considering each definition already grounded in the literature. That is, when the user made complaints about one of the factors, a complaint was counted. Table 2 shows, for each factor, examples of comments collected and amounts of related comments.

Table 3. Apps, their ratings and claims quantities by engagement factor
Full size table

Due to the limitation on the amount of data collected, we used five factors of engagement in our study: usability, endurability, control, confidence and efficiency - these factors were the most cited in complaints. The Table 3 provide data for each application, such as: rating and the number of complaints for each engagement factor checked. In the next Section, we present the analysis of our results.

5 Results

In order to verify how the engagement factors induce the evaluation of the users, we performed a linear correlation test evaluation. A numerical measure of linear correlation between two variables, takes values between −1 (perfectly strong and indirect relation) to +1 (perfectly strong and direct relation). Values close to zero indicate a lack of linear relationship. An array of these coefficients is called correlation matrix [38].

Fig. 2.
figure 2

Composition around the number of complaints of each engagement factor, per app. Rating in descending order.

Full size image
Fig. 3.
figure 3

Radar chart comprising multivariate data (engagement factors per app). Radar chart is used to measure scales, where each variable is “better” in some respect - in our case, in some engagement factor.

Full size image
Table 4. Engagement factors and their respective linear correlation values
Full size table

According to results presented in the Table 4, in Fig. 2 and in Fig. 3, we can observe that:

  • The endurability factor holds the greatest influence on the rating of the applications, presenting strong negative correlation.

  • The higher the number of complaints about the endurability, the lower the ratings assigned.

  • After the endurability factor, the usability, control and efficiency factors are the ones that most negatively influence the values assigned to the application ratings.

  • The two highest ratings have no complaints about usability and efficiency. The predominant complaints about the engagement factors are control and confidence.

  • The confidence factor has a positive correlation, where the higher the number of complaints, the higher the ratings. This fact can be seen as a consequence of the success of the application, since applications that hold larger numbers of users can get bigger problems around the confidence of user data.

  • In Fig. 3, through the multivariate comparative visualization of the radar chart, we can see that ratings applications with larger data points concentrate fewer complaints about the engagement factors. For example, the right-hand dimension of the radar chart has higher ratings and fewer demarcations than about general engagement factors, while the left-sided dimension - with smaller ratings apps - has a greater concentration of data points, represented by a greater variety of complaints about engagement factors.

6 Conclusions

Given the wide range of application options available on a variety of app store markets, it is still difficult to see what factors actually induce a good user evaluation of a particular app. The main objective, in this work, was to verify how negative comments, related to engagement factors, induce the users’ evaluation of some category of applications in a app store market - using the following hypothesis: the lower the number complaints, the higher the app rating.

Seven Android applications were selected in the Google Play Store. Each application was evaluated considering the following factors that influence engagement: usability, endurability, control, confidence and efficiency. The experimental results showed some interesting outcomes. Among them is the fact that the endurability factor has a greater negative influence on the rating of the applications than the other factors. That is, the greater the number of complaints of an application about endurability, the smaller the ratings. As a limitation, this work was carried out with a small sample of comments. As future work, we intend to use automated sentiment analysis on a higher data sample, and also use social networks to check for comments on the applications on different platforms.