Skip to main content
SpringerLink
Log in

Automated comprehensive evaluation approach for user interface satisfaction based on concurrent think-aloud method

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

The concurrent think-aloud protocol (CTA) is an effective method for collecting abundant product comments related to user satisfaction during the execution of evaluation tasks. However, manual analysis of these audio comments is time-consuming and labor-intensive. This paper aims to propose an approach for automated comprehensive evaluation of user interface (UI) satisfaction. It takes advantage of text mining and sentiment analysis (SA) techniques instead of manual analysis in order to assess user comments collected by the CTA. Based on the results of the SA, the proposed approach makes use of the analytic hierarchy process (AHP) method to evaluate the overall satisfaction and support developers for UI design improvements. In order to enhance the objectivity of evaluation, a sentiment matrix originating from text mining and SA on user comments is used to replace the criteria and the relative weights of the AHP method which were previously defined by experts. A comparison between the questionnaire survey method and the proposed approach in the empirical study suggested that the latter can efficiently evaluate UI satisfaction with high accuracy and provide designers abundant and specific information directly related to defects in design. It is argued that the proposed approach could be used as an automated framework for handling any type of comments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. http://www.keenage.com/html/e_index.html

  2. https://www.w3.org/TR/CSS/

  3. http://www.voicecloud.cn/

  4. http://nlp.stanford.edu/software/segmenter.shtml

References

  1. Frøkjær, E., Hertzum, M., Hornbæk, K.: Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? In: CHI 00 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 345–352 (2000)

  2. Calleros, J.M.G., Garca, J.G., Vanderdonckt, J.: Advance human machine interface automatic evaluation. Univers. Access Inf. Soc. 12(4), 387–401 (2013)

    Article  Google Scholar 

  3. Whitefield, A., Wilson, F., Dowell, F.: A framework for human factors evaluation. Behav. Inf. Technol. 10(1), 65–79 (1991)

    Article  Google Scholar 

  4. Davids, M.R., Chikte, U.M., Halperin, M.L.: Effect of improving the usability of an e-learning resource: a randomized trial. Adv. Physiol. Educ. 38(2), 155–60 (2014)

    Article  Google Scholar 

  5. Pinelle, D., Wong, N., Stach, T.: Heuristic evaluation for games: usability principles for video game design. In: CHI 13 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1453–1462 (2008)

  6. Grigoreanu, V., Mohanna, M.: Informal cognitive walkthroughs (ICW): paring down and pairing up for an agile world. In: CHI 13 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3093–3096 (2013)

  7. Schoonewille, H.H., Heijstek, W., Chaudron, M.R.V.: A cognitive perspective on developer comprehension of software design documentation. In: SIGDOC 11 Proceedings of the 29th ACM International Conference on Design of Communication, PP. 211–218 (2011)

  8. Nielsen, J.: Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco (1995)

    MATH  Google Scholar 

  9. Cooke, L.: Assessing concurrent think-aloud protocol as a usability test method: a technical communication approach. IEEE Trans. Prof. Commun. 53(3), 202–215 (2010)

    Article  Google Scholar 

  10. Peute, L.W.P., Keizer, N.F.D., Jaspers, M.W.M.: The value of retrospective and concurrent think aloud in formative usability testing of a physician data query tool. J. Biomed. Inf. 55, 1–10 (2015)

    Article  Google Scholar 

  11. Stefano, F., Borsci, S., Stamerra, G.: Web usability evaluation with screen reader users: implementation of the partial concurrent thinking aloud technique. Cognit. Process. 11(3), 263–272 (2010)

    Article  Google Scholar 

  12. Ivory, M.Y., Hearst, M.A.: State of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33(4), 470–516 (2001)

    Article  Google Scholar 

  13. Lewis, C.H.: Using the thinking-aloud method in cognitive interface design. Research report RC-9265, IBM (1982)

  14. Ericsson, K.A., Simon, H.A.: Protocol analysis: verbal reports as data. J. Market. Res. 23(3), 381–423 (1986)

    Google Scholar 

  15. Boren, T., Ramey, J.: Thinking aloud: reconciling theory and practice. IEEE Trans. Prof. Commun. 43(3), 261–278 (2000)

    Article  Google Scholar 

  16. Hertzum, M., Hansen, K.D., Andersen, H.H.K.: Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload? Behav. Inf. Technol. 28(2), 165–181 (2009)

    Article  Google Scholar 

  17. Olmsted-Hawala, E.L., Murphy, E.D., Hawala, S., Ashenfelter, K.T.: Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. In: Proceedings of 28th International Conference on Human Factors in Computing Systems, PP. 2381–2390 (2010)

  18. Anderson, E.W.: Customer satisfaction and word of mouth. J. Serv. Res. 1(1), 5–17 (1998)

    Article  Google Scholar 

  19. Hu, N., Pavlou, P.A., Zhang, J.: Can online reviews reveal a product’s true quality? empirical findings and analytical modeling of online word-of-mouth communication. In: EC 06 Proceedings of the 7th ACM Conference on Electronic Commerce, PP. 324–330 (2006)

  20. Hedegaard, S., Simonsen, J.G.: Extracting usability and user experience information from online user reviews. In: CHI’13 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2089–2098 (2013)

  21. Hu, M., Liu, B.: Mining and summarizing customer reviews. In: KDD 04 Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 168–177 (2004)

  22. Wu, M., Wang, L., Li, M., Long, H.: An approach of product usability evaluation based on Web mining in feature fatigue analysis. Comput. Ind. Eng. 75(1), 230–238 (2014)

    Article  Google Scholar 

  23. Kang, D., Park, Y.: Review-based measurement of customer satisfaction in mobile service: sentiment analysis and VIKOR approach. Expert Syst. Appl. 41(4), 1041–1050 (2014)

    Article  Google Scholar 

  24. Arora, C., Sabetzadeh, M., Briand, L., Zimmer, F.: Automated checking of conformance to requirements templates using natural language processing. IEEE Trans. Softw. Eng. 41(10), 944–968 (2015)

    Article  Google Scholar 

  25. Saaty, T.L.: Decision making with the analytic hierarchy process. Int. J. Serv. Sci. 1(1), 83–98 (2008)

    Google Scholar 

  26. Saaty, T.L.: Relative measurement and its generalization in decision making why pairwise comparisons are central in mathematics for the measurement of intangible factors the analytic hierarchy/network process, Revista de la Real Academia de Ciencias Exactas, Fsicas y Naturales. Serie A. Matemticas, Vol. 102, No. 2, September 2008, pp. 251–318 (2008)

  27. Matsuda, M., Uesugi, Y., Nonaka, T., Hase, T.: Design method of UI of AV remote controller based on AHP. In: Proceedings of 26th IEEE International Conference on Consumer Electronics, pp. 1–2 (2008)

  28. Delice, E.K., Güngör, Z.: The usability analysis with heuristic evaluation and analytic hierarchy process. Int. J. Ind. Ergon. 39(6), 934–939 (2009)

    Article  Google Scholar 

  29. Albayrak, E., Erensal, Y.C.: Using analytic hierarchy process (AHP) to improve human performance: an application of multiple criteria decision making problem. J. Intell. Manuf. 15(4), 491–503 (2004)

    Article  Google Scholar 

  30. Heim, S.: The Resonant Interface: HCI Foundations for Interaction Design. Addison-Wesley Longman Publishing Co., Inc., Boston (2007)

    Google Scholar 

  31. MA, Y.H., Wang, Y.C., SU, G.Y., Zhang, Y.M.: A novel Chinese text subject extraction method based on character co-occurrence. J. Comput. Res. Dev. 40(6), 874–878 (2003)

    Google Scholar 

  32. GE, B., Li, F., Guo, S., Tang, D.: Word s semantic similarity computation method based on Hownet. Appl. Res. Comput. 27(9), 3329–3333 (2010)

    Google Scholar 

  33. Freeman, B.: Triggered think-aloud protocol: using eye tracking to improve usability test moderation, In: CHI 11 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1171–1174 (2011)

  34. Vanderdonckt, J., Beirekdar, A.: Automated web evaluation by guideline review. J. Web Eng. Arch. 4(2), 102–117 (2005)

    Google Scholar 

Download references

Acknowledgements

We would like to thank the anonymous interviewees and reviewers for their detailed and insightful comments. This research work has been partially supported by the Supporting Programs of Science and Technology Department of SiChuan Province of China Grants, Nos. 2013GZX0138, 2014GZ0063.

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, Sichuan, China

    Weipeng Chen, Tao Lin, Li Chen & Peisa Yuan

Authors
  1. Weipeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tao Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Peisa Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tao Lin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, W., Lin, T., Chen, L. et al. Automated comprehensive evaluation approach for user interface satisfaction based on concurrent think-aloud method. Univ Access Inf Soc 17, 635–647 (2018). https://doi.org/10.1007/s10209-018-0610-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-018-0610-z

Keywords

Search

Navigation