Skip to main content
SpringerLink
Log in

Evaluating online learning communities

  • Research Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

This article reviews recent evaluation studies of online learning communities to provide a systematic understanding of how such communities are evaluated. Forty-two representative studies were selected and categorized into a newly developed taxonomy of online learning community evaluations. This taxonomy is divided into four components: evaluation purpose, evaluation approach, measures for evaluation, and evaluation techniques. The findings suggest that it is inappropriate to conceptualize evaluation of such communities as a one-size-fits-all, generalizable measure of “good” or “bad.” Instead, we recommend a comprehensive, on-going, diagnostic approach to measuring clusters of indicators, or syndromes, of a particular OLC and examining the causal relation assumed by the evaluators between what is measured and the success of OLC as an imputed outcome.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. e.g., Educational Technology Research & Development, Journal of Educational Computing Research, British Journal of Educational Technology, Computers & Education, Instructional Science, Journal of Interactive Learning Research, Journal of Asynchronous Learning Networks, Journal of Distance Education, etc.

  2. e.g., Proceeding of Computer Support for Collaborative Learning, Association for Educational Communications and Technology International Convention, International Conference of Learning Science, etc.

References

  • Alem, L., & Kravis, S. (2004). Design and evaluation of an online learning community: A case study at CSIRO. SIGGROUP Bulletin, 25(1), 20–24.

    Google Scholar 

  • Ardichvili, A., Page, N., & Wentling, T. (2002). Virtual knowledge-sharing communities of practice at caterpillar: Success factors and barriers. Performance Improvement Quarterly, 15(3), 94–113.

    Google Scholar 

  • Auyeung, L. H. (2004). Building a collaborative online learning community: A case study in Hong Kong. Journal of Educational Computing Research, 31(2), 119–136. doi:10.2190/YCM8-XKDY-QWN2-GPEH.

    Article  Google Scholar 

  • Aviv, R., Erlich, Z., Ravid, G., & Geva, A. (2003). Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks, 7(3), 1–23.

    Google Scholar 

  • Bielaczyc, K. (2001). Designing social infrastructure: The challenge of building computer-supported learning communities. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), European perspectives on computer-supported collaborative learning: Proceedings of the 1st European conference on computer-supported collaborative learning (pp. 106–114). Maastricht: Maastricht University.

  • Bozik, M., & Tracey, K. (2002). Fostering intellectual development in a learning community: Using an electronic discussion board. In P. Comeaux (Ed.), Communication and collaboration in the online classroom: Examples and applications (pp. 207–225). Boston, MA: Anker Publishing.

    Google Scholar 

  • Brett, C., Woodruff, E., & Nason, R. (1999). Online community and preservice teachers’ conceptions of learning mathematics. In C. M. Hoadley & J. Roschelle (Eds.), Proceedings of the computer support for collaborative learning (CSCL) (pp. 57–66). Mahwah, NJ: Erlbaum.

  • Brown, A. L., Ellery, S., & Campione, J. C. (1998). Creating zones of proximal development electronically. In J. G. Greeno & S. V. Goldman (Eds.), Thinking practices in mathematics and science learning (pp. 341–367). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Bruckman, A. (2004). Co-evolution of technological design and pedagogy in an online learning community. In S. A. Barab, R. Kling, & J. Gray (Eds.), Designing virtual communities in the service of learning (pp. 239–255). New York: Cambridge University Press.

    Google Scholar 

  • Bryce-Davis, H. (2001). Virtual learning communities. Paper presented at the Multimedia in the Home Conference, TRLabs, August 22–24, 2002, Saskatoon, Saskatchewan.

  • Carlen, U., & Jobring, O. (2005). The rationale of online learning communities. International Journal of Web Based Communities, 1(3), 272–295.

    Article  Google Scholar 

  • Chang, C. (2003). Towards a distributed web-based learning community. Innovations in Education and Teaching International, 40(1), 27–42.

    Article  Google Scholar 

  • Cho, H., Gay, G., Davidson, B., & Ingraffea, A. (2007). Social networks, communication styles, and learning performance in a CSCL community. Computers & Education, 49(2), 309–329.

    Article  Google Scholar 

  • Collins, T., Mulholland, P., & Watt, S. (2001). Using genre to support active participation in learning communities. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), European perspectives on computer-supported collaborative learning: Proceedings of the 1st European conference on computer-supported collaborative learning (pp. 156–164). Maastricht: Maastricht University.

  • Conrad, D. (2002). Deep in the hearts of learners: Insights into the nature of online community. Journal of Distance Education, 17(1), 1–19.

    Google Scholar 

  • Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Cuthell, J., Limited, A., Mills, R., & Derbyshire, I. (2002). MirandaNet: A learning community––a community of learners. Journal of Interactive Learning Research, 13(1/2), 167–186.

    Google Scholar 

  • De Laat, M., & Lally, V. (2003). Complexity, theory and praxis: Researching collaborative learning and tutoring processes in a networked learning community. Instructional Science, 31, 7–39.

    Article  Google Scholar 

  • Derry, S. J., & DuRussel, L. A. (1999, July). Assessing knowledge construction in on-line learning communities. Paper presented at the Annual Meeting of the International Society for Artificial Intelligence in Education, Lemans, France (ERIC Document Reproduction No. Ed 446897).

  • Engeström, Y. (1993). Developmental studies of work as a testbench of activity theory: Analyzing the work of general practitioners. In S. Chaiklin & J. Lave (Eds.), Understanding practice: Perspectives on activity and context (pp. 64–103). Cambridge: Cambridge University Press.

    Google Scholar 

  • Fusco, J., Gehlbach, H., & Schlager, M. (2000). Assessing the impact of a large-scale online teacher professional development community. Society for information technology and teacher education international conference 2000 (Vol. 1, pp. 2178–2183). Chesapeake, VA: AACE.

  • Haythornthwaite, C., Kazmer, M., Robins, J., & Shoemaker, S. (2000). Community development among distance learners: Temporal and technological dimensions. Journal of Computer-Mediated Communication, 6(1), Retrieved January 17, 2005, from http://www.ascusc.org/jcmc/vol6/issue1/haythornthwaite.html.

  • Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 117–136). Berlin: Springer-Verlag.

    Google Scholar 

  • Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Technology Research and Development, 48(3), 23–48.

    Article  Google Scholar 

  • Hiltz, S. R., & Turoff, M. (2002). What makes learning networks effective? Communications of the ACM, 45(4), 56–59.

    Article  Google Scholar 

  • Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.), Computer support for collaborative learning 2002 (pp. 453–462). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Hugo, J. (2002). Learning community history. New Directions for Adult and Continuing Education, 95, 5–26.

    Article  Google Scholar 

  • Hung, D., & Chen, D. (2002). Understanding how thriving Internet quasi-communities work: Distinguishing between learning about and learning to be. Educational Technology, 42(1), 23–27.

    Google Scholar 

  • Johnson, C. M. (2001). A survey of current research on online communities of practice. Internet and Higher Education, 4, 45–60.

    Article  Google Scholar 

  • Johnson, S. D., Suriya, C., Yoon, S. W., Berrett, J. V., & Fleur, J. L. (2002). Team development and group process of virtual learning teams. Computers & Education, 39, 379–393.

    Article  Google Scholar 

  • Ke, F., & Carr-Chellman, A. (2006). Solitary learner in online collaborative learning: A disappointing experience? Quarterly Review of Distance Education, 7(3), 249–265.

    Google Scholar 

  • Kilner, P. G., & Hoadley, C. M. (2005). Anonymity options and professional participation in an online community of practice. In T. Koschmann, T. W. Chan, & D. D. Suthers (Eds.), Proceedings of the computer support for collaborative learning (CSCL) 2005 (pp. 272–280). Mahwah, NJ: Erlbaum.

  • Kowch, E., & Schwier, R. (1997). Considerations in the construction of technology-based virtual learning communities. Canadian Journal of Educational Communication, 26(1), 1–12.

    Google Scholar 

  • Kuutti, K. (1996). Activity theory as a potential framework for human-computer interaction research. In B. A. Nardi (Ed.), Context and consciousness: Activity theory and human-computer interaction (pp. 17–44). Cambridge, MA: MIT Press.

    Google Scholar 

  • Lally, V., & Barrett, E. (1999). Building a learning community on-line: Towards socio-academic interaction. Research Papers in Education, 14(2), 147–163.

    Article  Google Scholar 

  • Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press.

    Google Scholar 

  • Leh, A. S. (2002). Action research on hybrid courses and their online communities. Educational Media International, 39(1), 31–38.

    Article  Google Scholar 

  • Linn, M. C., & Burbules, N. C. (1993). Construction of knowledge and group learning. In K. G. Tobin (Ed.), The practice of constructivism in science education (pp. 91–119). Washington, DC: American Association for the Advancement of Science (AAAS) Press.

    Google Scholar 

  • Littleton, K., & Whitelock, D. (2005). The negotiation and co-construction of meaning and understanding within a postgraduate online learning community. Learning, Media and Technology, 30(2), 147–164.

    Article  Google Scholar 

  • Lock, J. (2002). Laying the groundwork for the development of learning communities within online courses. Quarterly Review of Distance Education, 3, 395–408.

    Google Scholar 

  • Majdalani, M. C., Hollon, R. E., & McIntyre, S. R. (1999, February). The virtual learning technology community: Creating and sustaining professional development for K-16 learning communities. Paper presented at 10th Society for Information Technology & Teacher Education International Conference, Antonio, TX (ERIC Document Reproduction Service No. ED 432299).

  • Maor, D. (2003). The teacher’s role in developing interaction and reflection in an online learning community. Educational Media International, 40(1/2), 127–137.

    Article  Google Scholar 

  • McMillan, D. W., & Chavis, D. M. (1986). Sense of community: A definition and theory. Journal of Community Psychology, 14(1), 6–23.

    Article  Google Scholar 

  • McPherson, M., & Nunes, M. B. (2004). The failure of a virtual social space (VSS) designed to create a learning community: Lessons learned. British Journal of Educational Technology, 35(3), 305–321.

    Article  Google Scholar 

  • Mercer, N. (1995). The guided construction of knowledge: Talk amongst teachers and learners. Clevedon: Multilingual Matters.

    Google Scholar 

  • Meyers, R., Davis, H., & Botti, J. (2002, June). Professional development: Building effective virtual communities through cooperative learning. Paper presented at National Educational Computing Conference 2002, San Antonio, Texas (ERIC Document Reproduction Service No. ED 475943).

  • Miller, R. A. (1994). Medical diagonostic decision support systems––past, present, and future. Journal of the American Medical Informatics Association, 1(1), 8–27.

    Google Scholar 

  • Moore, K., Zaff, J., & Hair, E. (2002). Research and evaluation to serve the public: A commentary on Catalana, Berglund, Ryan, Lonczak, and Hawkins. Prevention & Treatment, 5(1).

  • Oliver, M. (2000). An introduction to the evaluation of learning technology. Educational Technology & Society, 3(4), 20–30.

    Google Scholar 

  • Oren, A., Mioduser, D., & Nachmias, R. (2002). The development of social climate in virtual learning discussion groups. International Review of Research in Open and Distance learning, 3(1). Retrieved, April 18, 2005, from http://www.icaap.org/iuicode?149.3.1.x.

  • Orey, M., Koenecke, L., & Crozier, J. (2003). Learning communities via the Internet a la epic learning: You can lead the horses to water, but you cannot get them to drink. Innovations in Education and Teaching International, 40(3), 260–269.

    Article  Google Scholar 

  • Palloff, R. M., & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass Publishers.

    Google Scholar 

  • Patton, M. (1997). Utilization-focused evaluation. London: Sage.

    Google Scholar 

  • Pearson, J. (1998). Electronic networking in initial teacher education: Is a virtual faculty of education possible? Computers & Education, 32, 221–238.

    Article  Google Scholar 

  • Poole, D. M. (2000). Student participation in a discussion-oriented online course: A case study. Journal of Research on Computing in Education, 33(2), 162–177.

    Google Scholar 

  • Preece, J. (2000). Sociability and usability in online communities: Determining and measuring success. Behaviour & Information Technology, 20(5), 347–356.

    Article  Google Scholar 

  • Renninger, K., & Shumar, W. (2002). Building virtual communities: Learning and change in cyberspace. New York: Cambridge University Press.

    Google Scholar 

  • Rheingold, H. (2000). The virtual community: Homesteading on the electronic frontier. Cambridge, MA: The MIT Press.

    Google Scholar 

  • Rice-Lively, M. L. (1994). Wired warp and woof: An ethnographic study of a networking class. Internet Research, 4(4), 20–35.

    Article  Google Scholar 

  • Ricketts, J., Wolfe, F. H., Norvelle, E., & Carpenter, E. H. (2000). Multimedia: Asynchronous distributed education––a review and case study. Social Science Computer Review, 18(2), 132–146.

    Article  Google Scholar 

  • Romiszowsky, A., & Mason, R. (1996). Computer-mediated communication. In D. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 438–456). New York: Simon & Schuster Macmillan.

    Google Scholar 

  • Rovai, A. P. (2001). Building classroom community at a distance: A case study. Educational Technology Research and Development, 49(4), 33–48.

    Article  Google Scholar 

  • Rovai, A. P. (2002). Sense of community, perceived cognitive learning, and persistence in aynchronous learning networks. The Internet and Higher Education, 5(4), 319–332.

    Article  Google Scholar 

  • Russell, M., & Ginsburg, L. (1999). Learning online: Extending the meaning of community: A review of three programs from the Southeastern United States (Report No. NCAL-TR99-01). National Center for Adult Literacy, Philadelphia, PA. SouthEast and Islands Regional Technology in Education Consortium, Greensboro, NC (ERIC Document Reproduction Service No. ED 437540).

  • Scardamalia, M., Bereiter, C., Brett, C., Burtis, P. J., Calhoun, C., & Smith Lea, N. (1992). Educational applications of a networked communal database. Interactive Learning Environments, 2(1), 45–71.

    Article  Google Scholar 

  • Schön, D. (1983). The reflective practitioner. New York: Basic Books.

    Google Scholar 

  • Scriven, M. (1991). Prose and cons about goal-free evaluation. American Journal of Evaluation, 12(1), 55–62.

    Article  Google Scholar 

  • Shea, P., Swan, K., Li, C. S., & Pickett, A. (2005). Developing learning community in online asynchronous college course: The role of teaching presence. Journal of Asynchronous Learning Networks, 9(4), 59–82.

    Google Scholar 

  • Soller, A., & Lesgold, A. (2003). A computational approach to analyzing online knowledge sharing interaction. In H. U. Hoppe, M. F. Verdejo, & J. Kay (Eds.), Artificial intelligence in education (pp. 253–260). Amsterdam, Netherlands: IOS Press.

    Google Scholar 

  • Spitzer, W., & Wedding, K. (1995). Labnet: An international electronic community for professional development. Computer Education, 24(3), 247–255.

    Article  Google Scholar 

  • Squire, K. D., & Johnson, C. B. (2000). Supporting distributed communities of practice with interactive television. Educational Technology Research and Development, 48(1), 23–43.

    Article  Google Scholar 

  • Steinkuehler, C. A. (2004). Learning in massively multiplayer online games. In Proceedings of the 6th international conference on learning sciences, Los Angeles, California (pp. 521–528).

  • Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park: Sage.

    Google Scholar 

  • Swan, K. (2002). Building learning communities in online courses: The importance of interaction. Education, Communication & Information, 2(1), 23–49.

    Article  Google Scholar 

  • Thomas, M. J. W. (2002). Learning within incoherent structures: The space of online discussion forums. Journal of Computer Assisted Learning 18(3): 351–366.

    Google Scholar 

  • Tu, C. H., & Corry, M. (2002). ELearning communities. The Quarterly Review of Distance Education, 3(2), 207–218.

    Google Scholar 

  • Upcraft, M. L., & Schuh, J. H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.

    Google Scholar 

  • Wang, M., Sierra, C., & Folger, T. (2003). Building a dynamic online learning community among adult learners. Education Media International, 40(1/2), 49–61.

    Article  Google Scholar 

  • Watkins, D. S. (1997). The development, pilot test, assessment, and evaluation of a computerized online internet community system at programs for higher education. Unpublished doctoral dissertation, Nova Southeastern University, Fort Lauderdale.

  • Wegerif, R. (1998). The social dimension of asynchronous learning networks, Journal of Asynchronous Learning Networks, 2(1). Retrieved April 15, 2004, from http://www.aln.org/alnweb/journal/vo12issue1/wegerif.htm.

Download references

Author information

Authors and Affiliations

  1. College of Education, Organizational Learning and Instructional Technology Program, University of New Mexico, 294 Hokona Hall, MSC05 3040, Albuquerque, NM, 87131-1231, USA

    Fengfeng Ke

  2. Steinhardt School of Culture, Education, and Human Development, New York University, New York, NY, USA

    Christopher Hoadley

Authors
  1. Fengfeng Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christopher Hoadley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fengfeng Ke.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ke, F., Hoadley, C. Evaluating online learning communities. Education Tech Research Dev 57, 487–510 (2009). https://doi.org/10.1007/s11423-009-9120-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-009-9120-2

Keywords

Search

Navigation