Skip to main content
SpringerLink
Log in

The Crowd Thinks Aloud: Crowdsourcing Usability Testing with the Thinking Aloud Method

  • Conference paper
  • First Online:
HCI International 2021 - Late Breaking Papers: Design and User Experience (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 13094))

Included in the following conference series:

Abstract

Thinking aloud (TA) is a widely employed usability testing method and allows identifying usability problems based on testers’ verbalization. Conducting TA studies is laborious and expensive due to the recruiting of participants, the required workforce and facilities, and the resulting monetary costs. Also, it is usually carried out in very controlled settings like laboratories, thereby limiting participants’ diversity, test realism, and the quality of the results. Crowdsourcing, in contrast, provides access to a wide and diverse workforce at moderate costs. Hence, it might help to overcome the limitations of traditional TA. Although usability testing has been successfully crowdsourced with other methods before, there is still no evidence on the feasibility of conducting TA involving unsupervised workers as testers. Thus, we conducted a between-subjects user study using the example of an online web page, both via a traditional remote TA setup and a crowdsourcing setup with a self-developed platform. The results do not show a significant difference between both setups regarding participants’ performance, and the quality and quantity of the identified usability problems. Therefore, we conclude that crowdsourcing is a feasible and cost-effective solution to conduct usability testing with the TA method, at least for selected use cases.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.webex.com/ Accessed June 2021.

  2. 2.

    https://www.edx.org/ Accessed June 2021.

  3. 3.

    https://github.com/nam-tuilmenau/crowd-ta.

  4. 4.

    https://www.microworkers.com/ Accessed June 2021.

References

  1. Alhadreti, O., Mayhew, P.: To intervene or not to intervene: an investigation of three think-aloud protocols in usability testing. J. Usability Stud. 12(3), 111–132 (2017)

    Google Scholar 

  2. Behrend, T.S., Sharek, D.J., Meade, A.W., Wiebe, E.N.: The viability of crowdsourcing for survey research. Behav. Res. Methods 43(3), 800–813 (2011). https://doi.org/10.3758/s13428-011-0081-0

    Article  Google Scholar 

  3. Bruggemann, J., Lander, G.C., Su, A.I.: Exploring applications of crowdsourcing to cryo-EM. J. Struct. Biol. 203(1), 37–45 (2018). https://doi.org/10.1016/j.jsb.2018.02.006

    Article  Google Scholar 

  4. Denning, S., Hoiem, D., Simpson, M., Sullivan, K.: The value of thinking-aloud protocols in industry: a case study at microsoft corporation. Proc. Hum. Factors Soc. Ann. Meet. 34(17), 1285–1289 (1990). https://doi.org/10.1177/154193129003401723

    Article  Google Scholar 

  5. Egger-Lampl, S., et al.: Crowdsourcing quality of experience experiments. In: Archambault, D., Purchase, H., Hoßfeld, T. (eds.) Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments, LNCS, vol. 10264, pp. 154–190. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66435-4_7

  6. Ericsson, K.A., Simon, H.A.: Protocol Analysis: Verbal Reports as Data. The MIT Press, Cambridge (1984)

    Google Scholar 

  7. Estellés-Arolas, E.L., Guevara, F.G., Towards an integrated crowdsourcing definition: Towards an integrated crowdsourcing definition. J. Inf. Sci. 38, 189–200 (2012). https://doi.org/10.1177/0165551512437638

    Article  Google Scholar 

  8. Fan, M., Shi, S., Truong, K.N.: Practices and challenges of using think-aloud protocols in industry: an international survey. J. Usability Stud. 15(2) (2020)

    Google Scholar 

  9. Haak, M.J., Jong, M.D., Schellens, P.J.: Evaluating municipal websites: a methodological comparison of three think-aloud variants. Gov. Inf. Qual. 26(1) (2009). https://doi.org/10.1016/j.giq.2007.11.003

  10. Hirth, M., Hoßfeld, T., Tran-Gia, P.: Anatomy of a crowdsourcing platform - using the example of Microworkers.com. In: International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (2011). https://doi.org/10.1109/IMIS.2011.89

  11. Hossfeld, T., et al.: Best practices for QOE crowdtesting: QOE assessment with crowdsourcing. IEEE Trans. Multimedia 16(2), 541–558 (2014). https://doi.org/10.1109/TMM.2013.2291663

    Article  Google Scholar 

  12. Kittur, A., Chi, E.H., Suh, B.: Crowdsourcing user studies with mechanical Turk. In: Conference on Human Factors in Computing Systems (2008). https://doi.org/10.1145/1357054.1357127

  13. Krug, S.: Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems. New Riders (2009)

    Google Scholar 

  14. Krug, S.: Don’t Make Me Think. Revisited - A Common Sense Approach to Web Usability, New Riders (2014)

    Google Scholar 

  15. Lewis, C.: Using the “Thinking Aloud’’ Method in Cognitive Interface Design. IBM TJ Watson Research Center Yorktown Heights, NY (1982)

    Google Scholar 

  16. Liu, D., Bias, R.G., Lease, M., Kuipers, R.: Crowdsourcing for usability testing. In: Proceedings of the American Society for Information Science and Technology 49(1) (2012). https://doi.org/10.1002/meet.14504901100

  17. Maier-Hein, L., et al.: Can masses of non-experts train highly accurate image classifiers? a crowdsourcing approach to instrument segmentation in laparoscopic images. Med. Image Comput. Comput.-Assist. Interv. - MICCAI 2014(17), 438–445 (2014). https://doi.org/10.1007/978-3-319-10470-6_55

    Article  Google Scholar 

  18. Meier, F.: Crowdsourcing als Rekrutierungsstrategie im Asynchronen Remote-Usability-Test. Information-Wissenschaft und Praxis, 63(5) (2012). https://doi.org/10.1515/iwp-2012-0063

  19. Nebeling, M., Speicher, M., Norrie, M.C.: CrowdStudy: general toolkit for Crowdsourced evaluation of web interfaces. In: SIGCHI Symposium on Engineering Interactive Computing Systems (2013). https://doi.org/10.1145/2494603.2480303

  20. Nielsen, J.: Usability Engineering. Elsevier Science (1994)

    Google Scholar 

  21. Nielsen, J.: Estimating the number of subjects needed for a thinking aloud test. Int. J. Hum. - Comput. Stud. 41(3) (1994)

    Google Scholar 

  22. Nielsen, L., Chavan, S.: Differences in task descriptions in the think aloud test. In: Aykin, N. (ed.) Usability and Internationalization. HCI and Culture, UI-HCII 2007. LNCS, vol. 4559, pp. 174–180. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73287-7_22

  23. Rosenthal, R.: Meta-Analytic Procedures for Social Research. SAGE Publications, Inc. (1991)

    Google Scholar 

  24. Rubin, J., Chisnell, D., Spool, J.: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. Wiley, Hoboken (2011)

    Google Scholar 

  25. Schneider, H., Frison, K., Wagner, J., Butz, A.: CrowdUX: a case for using widespread and lightweight tools in the quest for UX. In: Conference on Designing Interactive Systems (2016). https://doi.org/10.1145/2901790.2901814

  26. Skov, M.B., Stage, J.: Supporting problem identification in usability evaluations. In: Australia Conference on Computer-Human Interaction (2005)

    Google Scholar 

  27. Sova, D.H., Nielsen, J.: 234 Tips and Tricks for Recruiting Users as Participants in Usability Studies. Nielsen N Group (2003)

    Google Scholar 

  28. Thompson, K.E., Rozanski, E.P., Haake, A.R.: Here, there, anywhere: remote usability testing that works. In: Conference on Information Technology Education (2004). https://doi.org/10.1145/1029533.1029567

  29. Vermeeren, A.P.O.S., den Bouwmeester, K., Aasman, J., Ridder, H., de Ridder, H.: DEVAN: a tool for detailed video analysis of user test aata. Behav. IT 21(6) (2002)

    Google Scholar 

  30. Yuhui, W., Tian, L., Xinxiong, L.: Reliability of perceived usability assessment via crowdsourcing platform: retrospective analysis and novel feedback quality inspection method. Int. J. Hum.-Comput. Int. 36(11) (2020). https://doi.org/10.1080/10447318.2019.1709339

Download references

Author information

Authors and Affiliations

  1. User-Centric Analysis of Multimedia Data Group, TU Ilmenau, Ilmenau, Germany

    Edwin Gamboa, Rahul Galda & Matthias Hirth

  2. u.works GmbH, Ilmenau, Germany

    Cindy Mayas

Authors
  1. Edwin Gamboa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rahul Galda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cindy Mayas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matthias Hirth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Edwin Gamboa .

Editor information

Editors and Affiliations

  1. University of Creteand Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

  2. Department of Design, Federal University of Pernambuco, Recife, Brazil

    Marcelo M. Soares

  3. World Usability Day and Brandeis University, Newton Center, MA, USA

    Elizabeth Rosenzweig

  4. Aaron Marcus and Associates, Berkeley, CA, USA

    Aaron Marcus

  5. Tokyo University of Science, Tokyo, Saitama, Japan

    Sakae Yamamoto

  6. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

  7. Tsinghua University, Beijing, China

    Pei-Luen Patrick Rau

  8. Computer and Information Sciences, Towson University, Towson, MD, USA

    Gabriele Meiselwitz

  9. DePaul University, Chicago, IL, USA

    Xiaowen Fang

  10. San Jose State University, San Jose, CA, USA

    Abbas Moallem

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gamboa, E., Galda, R., Mayas, C., Hirth, M. (2021). The Crowd Thinks Aloud: Crowdsourcing Usability Testing with the Thinking Aloud Method. In: Stephanidis, C., et al. HCI International 2021 - Late Breaking Papers: Design and User Experience. HCII 2021. Lecture Notes in Computer Science(), vol 13094. Springer, Cham. https://doi.org/10.1007/978-3-030-90238-4_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90238-4_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90237-7

  • Online ISBN: 978-3-030-90238-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation