Abstract
This paper reports the results of a three-stage usability test of a modeling tool designed to support learners’ deep understanding of the impacts of climate change on ecosystems. The design process involved repurposing an existing modeling technology used by professional scientists into a learning tool specifically designed for middle school students. To evaluate usability, we analyzed students’ task performance and task completion time as they worked on an activity with the repurposed modeling technology. In stage 1, we conducted remote testing of an early modeling prototype with urban middle school students (n = 84). In stages 2 and 3, we used screencasting software to record students’ mouse and keyboard movements during collaborative think-alouds (n = 22) and conducted a qualitative analysis of their peer discussions. Taken together, the study findings revealed two kinds of usability issues that interfered with students’ productive use of the tool: issues related to the use of data and information, and issues related to the use of the modeling technology. The study findings resulted in design improvements that led to stronger usability outcomes and higher task performance among students. In this paper, we describe our methods for usability testing, our research findings, and our design solutions for supporting students’ use of the modeling technology and use of data. The paper concludes with implications for the design and study of modeling technologies for science learning.
Similar content being viewed by others
References
Alibrandi M (2003) GIS in the classroom: using geographic information systems in social studies and environmental science. Heinemann, Portsmouth
Baker TR (2005) Internet-based GIS mapping in support of K-12 education. Prof Geogr 57(1):44–50
Baker TR, White SH (2003) The effects of G.I.S. on students’ attitudes, self-efficacy, and achievement middle school science classrooms. J Geogr 102(6):243–254
Bausmith JM, Leinhardt G (1998) Middle-school students’ map construction: understanding complex spatial displays. J Geogr 97(3):93–107
Beach JH, Stewart AM, Vorontsov, GY (2002) Mapping distributed life with distributed computation. In: Proceedings of the 22nd annual Esri international user conference. Redlands, CA: Esri Publications
Benyon D (1993) Adaptive systems: a solution to usability problems. User Model User Adap Inter 3(1):65–87
Board College (2009) Science: college board standards for college success. The College Board Press, New York
Bury KF (1984) The iterative development of usable computer interfaces. In Proceedings of IFIP INTERACT’84 international conference on human–computer interaction, pp 743–748. London, UK, Sept 4–7
Ceaparu I, Lazar J, Bessiere K, Robinson J, Schneiderman B (2004) Determining causes and severity of end-user frustration. Int J Hum Comput Interact 17(3):333–356
Coll R, France B, Taylor I (2005) The role of models/and analogies in science education: implications from research. Int J Sci Edu 27(2):183–198
Collins A (2011) A study of expert theory formation: the role of different model types and domain frameworks. In: Khine MS, Saleh IM (eds) Models and modeling: cognitive tools for scientific enquiry. Springer, New York, pp 23–40
Crowther MS, Keller CC, Waddoups GL (2004) Improving the quality and effectiveness of computer-mediated instruction through usability evaluations. British J Edu Technol 35(3):289–303
Csikszentmihalyi M (1997) Finding flow: the psychology of engagement with everyday life. Basic Books, New York
Edelson DC (2004). Designing GIS software for education: a workshop report for the GIS community. The Geographic Data in Education Initiative at Northwestern University
Edelson DC, Gordin D (1998) Visualization for learning: a framework for adapting scientist’s tools. Comput Geosci 24(7):607–616
Gobert JD, Pallant A (2004) Fostering students’ epistemologies of models via authentic model-based tasks. J Sci Educ Technol 13(1):7–22
Hewitt J, Brett C (2011) Engaging learners in the identification of key ideas in complex online discussions. In Spada H, Stahl G, Miyake N, Law N (eds) In: Proceedings of the computer-supported collaborative learning conference, pp 960–961, Hong Kong, July 4–8
IPCC (2007) Climate change 2007: synthesis report. In: Core writing team, Pachauri RK, Reisinger A (eds) Contribution of working groups I, II and III to the fourth assessment report of the intergovernmental panel on climate change. IPCC, Geneva, Switzerland
Johnson RR, Salvo MJ, Zoetewey MW (2007) User-centered technology in participatory culture: two decades “Beyond a narrow conception of usability testing”. Proc IEEE Transact Prof Commun 50(4):320–332
Kaplan DE, Black JB (2003) Mental models and computer-based scientific inquiry learning: effects of mechanism cues on adolescent representation and reasoning about causal systems. J Sci Educ Technol 12(4):483–493
Keating T, Barnett MG, Barab SA, Hay KE (2002) The virtual solar system project: developing conceptual understanding of astronomical concepts through building three-dimensional computational models. J Sci Educ Technol 11(3):261–275
Landsberger HenryA (1958) Hawthorne revisited: management and the worker: its critics, and developments in human relations in industry. Cornell University Press, Ithaca
Lazar J, Jones A, Shneiderman B (2006) Workplace user frustration with computers: an exploratory investigation of the causes and severity. Behav Inf Technol 25(3):239–251
Lehrer R, Schauble L (2006) Scientific thinking and science literacy. In: Damon W, Lerner R, Renninger KA, Sigel IE (eds) Handbook of child psychology, vol 4, 6th edn. Wiley, Hoboken
Manouselis N, Drachsler H, Vuorikari R, Hummel H, Koper, K (2011) Recommender sytems in technology enhanced learning. In: Ricci F, Rokach L, Shapira B, Kantor PB (eds) Recommener systems handbook. Springer, New York, pp 387–415
Michigan Department of Education. K-7 Content grade level expectations in science, version 1.09. Retrieved from: http://michigan.gov/mde/
Miles MB, Huberman AM (1994) Qualitative data analysis: an expanded sourcebook, 2nd edn. Sage Publications, Thousand Oaks
Miyake N (1986) Constructive interaction and the iterative process of understanding. Cognitive Science 10:151–177
National Research Council (2012) A framework for K-12 science education: practices, crosscutting concepts, and core ideas. National Academies Press, Washington
Nelson BD, Aron RH, Francek MA (1992) Clarification of selected misconceptions in physical geography. J Geogr 91(2):76–80
Nielsen J (1993) Iterative user-interface design. IEEE Comput 26(11):32–41
Nielsen J (2003) Usability 101: an introduction to usability. Jakob Nielsen’s Alertbox. Base article retrieved 2 Nov 2010, from: http://www.useit.com/alertbox/20030825.html
Nielsen J (2010). Children’s websites: usability issues in designing for kids. Jakob Nielsen’s Alertbox. Article retrieved 2 Nov 2010, from: http://www.useit.com/alertbox/children.html
Nielsen J, Landauer TK (1993) A mathematical model of the finding of usability problems. In Proceedings of ACM INTERCHI’93 conference. ACM Press, Amsterdam, pp 206–213
Pallant A, Tinker RF (2004) Reasoning with atomic-scale molecular dynamic models. J Sci Educ Technol 13(1):51–66
Preece J, Rogers Y, Sharp H (2002) Interaction design: beyond human-computer interaction. Wiley, New York
Quintana C, Reiser BJ, Davis EA, Krajcik J, Fretz E, Duncan RG, Kyza E, Edelson D, Soloway E (2004) A scaffolding design framework for software to support science inquiry. J Learn Sci 13(3):337–386
Schwarz CV, White BY (2005) Metamodeling knowledge: developing students’ understanding of scientific modeling. Cogn Instr 32(2):165–205
Sengupta P, Wilensky U (2011) Lowering the learning threshold: multi-agent-based models and learning electricity (pp 141–171). In Khine MS, Saleh IM (eds) Models and modeling: cognitive tools for scientific enquiry. Springer, New York, pp 23–40
Shapiro AM (2008) Hypermedia design as learner scaffolding. Edu Tech Res Dev 56:29–44
Shneiderman B, Bederson BB (2005) Maintaining concentration to achieve task completion. In: Proceedings of the 2005 designing user experience conference. AIGA Press, San Francisco
Shorkey CT, Crocker SB (1981) Frustration theory: a source of unifying concepts for generalist practice. Soc Work 26(5):374–379
Sim G, MacFarlane S, Read J (2006) All work and no play: measuring fun, usability and learning in software for children. Comput Educ 46(3):235–248
Soloway E, Guzdial M, Hay KE (1994) Learner-centered design: the challenge for HCI in the 21st century. Interactions 1(2):36–48
Songer NB (2006) BioKIDS: an animated conversation on the development of complex reasoning in science (pp 355–369). In: Sawyer RK (ed) Cambridge handbook of the learning sciences. Cambridge University Press, New York
Songer NB, Kelcey B, Gotwals AW (2009) How and when does complex reasoning occur? Empirically driven development of a learning progression focused on complex reasoning about biodiversity. J Res Sci Teach 46(6):610–631
Sullivan P (1989) Beyond a narrow conception of usability testing. Proc IEEE Transac Prof Commun 32(4):256–264
Taylor I, Barker M, Jones A (2003) Promoting mental model building in astronomy education. Int J Sci Edu 25(10):1205–1225
Turner CW, Lewis JR, Nielsen J (2006) Determining usability test sample size. In: Karwowski W (ed) International encyclopedia of ergonomics and human factors, vol 4, 6th edn. CRC Press, Boca Raton
Uttal DH (2000) Seeing the big picture: map use and the development of special cognition. Dev Sci 3(3):247–264
Varma K, Linn MC (2011) Using interactive technology to support students’ understanding of the greenhouse effect and global warming. J Sci Educ Technol. doi:10.1007/s10956-011-9337-9
Virzi RA (1992) Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors 34(4):457–468
Acknowledgments
This research is based upon work supported by the National Science Foundation under Grant No. 0918590. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Peters, V.L., Songer, N.B. Evaluating the Usability of a Professional Modeling Tool Repurposed for Middle School Learning. J Sci Educ Technol 22, 681–696 (2013). https://doi.org/10.1007/s10956-012-9422-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10956-012-9422-8