Skip to main content
SpringerLink
Log in

Evaluation on crowdsourcing research: Current status and future direction

  • Published:
Information Systems Frontiers Aims and scope Submit manuscript

Abstract

Crowdsourcing is one of the emerging Web 2.0 based phenomenon and has attracted great attention from both practitioners and scholars over the years. It can facilitate the connectivity and collaboration of people, organizations, and societies. We believe that Information Systems scholars are in a unique position to make significant contributions to this emerging research area and consider it as a new research frontier. However, so far, few studies have elaborated what have been achieved and what should be done. This paper seeks to present a critical examination of the substrate of crowdsourcing research by surveying the landscape of existing studies, including theoretical foundations, research methods, and research foci, and identifies several important research directions for IS scholars from three perspectives—the participant, organization, and system—and which warrant further study. This research contributes to the IS literature and provides insights for researchers, designers, policy-makers, and managers to better understand various issues in crowdsourcing systems and projects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Agarwal, R., & Henry Lucas, C. J. (2005). The information systems identify crisis: Focusing on high-visibility and high-impact research. MIS Quarterly, 29(3), 381–398.

    Google Scholar 

  • Alavi, M., & Carlson, P. (1992). A review of MIS research and disciplinary development. Journal of Management Information Systems, 8(4), 45–62.

    Google Scholar 

  • Albors, J., Ramos, J. C., & Hervas, J. L. (2008). New learning network paradigms: communities of objectives, crowdsourcing, wikis and open source. International Journal of Information Management, 28(3), 194–202.

    Article  Google Scholar 

  • Alonso, O., Rose, D., & Stewart, B. (2008). Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2), 9–15.

    Article  Google Scholar 

  • Ambati, V., Vogel, S., & Carbonell, J. (2010). Active learning and crowd-sourcing for machine translation. Proceedings of the LREC, 2010, 2169–2174.

    Google Scholar 

  • Archak, N. (2010). Money, glory and cheap talk: Analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on TopCoder.com. in Proceedings of the 19th international conference on World Wide Web ACM, pp. 21–30.

  • Archak, N., & Sundararajan, A. (2009). “Optimal Design of Crowdsourcing Contest,” in Proceedings Thirtieth International Conference on Information Systems (ICIS 2009), Phoenix, USA.

  • Ba, S., Stallaert, J., & Whinston, A. B. (2001). Introducing a third dimension in decentralized information systems design: the case for incentive alignment. Information Systems Research, 12(3), 225–239.

    Article  Google Scholar 

  • Bagozzi, R. P., & Dholakia, U. M. (2006). Open source software user communities: a study of participation in Linux user groups. Management Science, 52(7), 1099–1115.

    Google Scholar 

  • Bayus, B. L. (2010). Crowdsourcing and individual creativity over time: the detrimental effects of past success. SSRN Quantitative Marketing eJournal, Retrieved from http://ssrn.com/abstract=1667101.

  • Benbasat, I., & Zmud, R. W. (2003). The identify crisis within the IS discipline: defining and communicating the discipline’s core properties. MIS Quarterly, 27(2), 183–194.

    Google Scholar 

  • Bigliardi, B., Dormio, A. I., & Galati, F. (2012). The adoption of open innovation within the telecommunication industry. European Journal of Innovation Management, 15(1), 27–54.

    Article  Google Scholar 

  • Bonabeau, E. (2009). Decisions 2.0: the power of collective intelligence. MIT Sloan Management Review, 50(2), 45–52.

    Google Scholar 

  • Boudreau, M., Gefen, D., & Straub, D. (2001). Validation in IS research: a state-of-the-art assessment. MIS Quarterly, 25(1), 1–24.

    Article  Google Scholar 

  • Boudreau, K. J., Nicola, L., & Karim, R. L. (2011). Incentives and problem uncertainty in innovation contests: an empirical Analysis. Management Science, 57(5), 843–863.

    Article  Google Scholar 

  • Brabham, D. C. (2008a). Crowdsourcing as a model for problem solving: an introduction and cases. Convergence: The International Journal of Research into New Media Technologies, 14(1), 75–90.

    Google Scholar 

  • Brabham, D. C. (2008b). “Moving the crowd at iStockphoto: the composition of the crowd and motivations for participation in a crowdsourcing application. First Monday, 13(6).

  • Brabham, D. C. (2009). Crowdsourcing the public participation process for planning projects. Planning Theory, 8(3), 242–262.

    Article  Google Scholar 

  • Brabham, D. C. (2010). Moving the crowd at threadless: motivations for participation in a crowdsourcing application. Information, Communication & Society, 13(8), 1122–1145.

    Article  Google Scholar 

  • Brabham, D. C. (2011) forthcoming. Crowdsourcing: A model for leveraging online communities. In A. Delwiche & J. Henderson (Eds.), The Routledge Handbook of Participatory Culture.

  • Brew, A., Greene, D., & Cunningham, P. (2010). Using crowdsourcing and active learning to track sentiment in online media. in Proceedings of ECAI 2010 19th European Conference on Artificial Intelligence, pp. 86–96.

  • Brito, J. (2008). Hack, Mash, & Peer: Crowdsourcing government transparency. The Columbia Science and Technology Law Review, 9, 119–157.

    Google Scholar 

  • Chanal, V., Caron-Fasan, & M. L. (2008). How to invent a new business model based on crowdsourcing: The CROWDSPIRIT Case. Conférence de l'Association Internationale de Management Stratégique, Nice, May.

  • Cheon, M. J., Grover, V., & Teng, T. C. (1995). Theoretical perspectives on the outsourcing of information systems. Journal of Information Technology, 10(4), 209–219.

    Article  Google Scholar 

  • Chesbrough, H. W. (2003). The era of open innovation. MIT Sloan Management Review, 44(3), 35–41.

    Google Scholar 

  • Chesbrough, H. (2006a). Open innovation: A new paradigm for understanding industrial innovation. In Chesbrough et al., 2006. “Open Innovation: Researching a New Paradigm (pp 1–12). Oxford: Oxford University Press.

  • Chesbrough, H. (2006b). Open business models: How to thrive in the new innovation landscape. Cambridge, MA: Harvard Business School Press.

    Google Scholar 

  • Chiaroni, D., Chiesa, V., & Frattini. (2009). Investigating the adoption of open innovation in the bio-pharmaceutical industry: a framework and an empirical analysis. European Journal of Innovation Management, 12(3), 285–305.

    Article  Google Scholar 

  • Choffnes, D. R., Bustamante, F. E., & Ge, Z. H. (2010). Crowdsourcing service-level network event monitoring. in Proceedings of the ACM SIGCOMM 2010 conference on SIGCOMM ACM, pp. 387–398.

  • Chris, C. B. (2009). “Fast, cheap, and creative: Evaluating translation quality using Amazons mechanical Turk,” in Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Vol. 1. PA, USA: Association for Computational Linguistics Stroudsburg

  • Delaney, D. G. (2008). Marine invasive species: validation of citizen science and implications for national monitoring networks. Biological Invasions, 10, 117–128.

    Article  Google Scholar 

  • Diana, R. (2010). Crowdsourcing is outsourcing Web 2.0 Style. Retrieved January 12, from http://regulargeek.com/2010/05/14/crowdsourcing-is-outsourcing-web-2-0-style/

  • DiPalantino, D., & Vojnovic, M. (2009). Crowdsourcing and all-pay auctions. in Proceedings of the Tenth ACM conference on Electronic commerce, pp. 119–128.

  • Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4), 86–96.

    Article  Google Scholar 

  • Eckert, K., Niepert, M., Niemann, C., Buckner, C., Allen, C., & Stuckenschmidt, H. (2010). Crowdsourcing the Assembly of Concept Hierarchies. in Proceedings of the 10th annual joint conference on Digital libraries ACM, pp. 139–148.

  • Estellés-Arolas, E., & González Ladrón-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science (in press)

  • Franklin, M, Kossman, D., Kraska, T, Ramesh, S, & Xin, R. (2011). CrowdDB: Answering Queries with Crowdsourcing. In Proceedings of SIGMOD 2011.

  • Gallivan, M. J. (2001). Striking a balance between trust anti control in a virtual organization: a content analysis of open source software case studies. Information Systems Journal, 11(4), 277–304.

    Article  Google Scholar 

  • Galloway, A. W. E., et al. (2006). The reliability of citizen science: a case study of Oregon white oak stand surveys. Wildlife Society Bulletin, 34, 1425–1429.

    Article  Google Scholar 

  • Greengard, S. (2011). Following the Crowd. Communications of the ACM, 54(2), 20–22.

    Article  Google Scholar 

  • Gregg, D. G. (2010). Designing for collective intelligence. Communications of the ACM, 53(4), 134–138.

    Article  Google Scholar 

  • Gregor, S. (2006). The Nature of Theory in Information Systems. MIS Quarterly, 30(3), 611–642.

    Google Scholar 

  • Hars, A., & Ou, S. S. (2002). Working for free? Motivations for participating in open-source projects. International Journal of Electronic Commerce, 6(3), 25–39.

    Google Scholar 

  • Hauser, L. (1988). A conceptual analysis of information science. Library and Information Science Research, 10, 3–34.

    Google Scholar 

  • Heer, J., & Bostock, M. (2010). Crowdsourcing Graphical Perception. in Proceedings of the 28th international conference on Human factors in computing systems ACM, pp. 286–295.

  • Hertel, G., Niedner, S., & Herrmann, S. (2003). Motivation of software developers in open source projects: an internet-based survey of contributors to the Linux Kernel. Research Policy, 32(7), 1159–1177.

    Google Scholar 

  • Hevner, A., March, S., Park, J., & Ram, S. (2004). Design science in information systems research. MIS Quarterly, 28(1), 75–105.

    Google Scholar 

  • Holley, R. (2009). Crowdsourcing and social engagement: potential, power and freedom for libraries and users. in Proceedings of Pacific Rim Digital Library Alliance (PRDLA) Annual meeting and Conference: Libraries at the End of the World: Digital Content and Knowledge Creation, pp. 1–28.

  • Horton, J. J., & Chilton, L. B. (2010). The Labor Economics of Paid Crowdsourcing. in Proceedings of the 11th ACM conference on Electronic commerce ACM, pp. 209–218.

  • Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), 1–4.

    Google Scholar 

  • Howe, J. (2008). Crowdsourcing. New York: Crown Publishing Group.

    Google Scholar 

  • Hsueh, P.-Y., Melville, P., & Sindhwani, V. (2009). Data quality from crowdsourcing: A study of annotation selection criteria. in Proceedings of the NAACL HLT 2009Workshop on Active Learning for Natural Language Processing, pp. 27–35.

  • Huberman, B. A., Romero, D., & Wu, F. (2009). Crowdsourcing, attention and productivity. Journal of Information Science, 35(6), 758–765.

    Article  Google Scholar 

  • Hudson-Smith, A., Batty, M., Crooks, A., & Milton, R. 2009. Mapping for the Masses: Accessing Web 2.0 through Crowdsourcing. Social Science Computer Review, pp. 1066–1084.

  • Ipeirotis, P. G., Provost, F., & Wang, J. (2010). Quality Management on Amazon Mechanical Turk. In KDD-HCOMP’10, July 2010, Washington DC, USA.

  • Jain, R. (2010). Investigation of Governance Mechanisms for Crowdsourcing Initiatives. in Proceedings of Americas Conference on Information Systems, pp. 557–563.

  • Johnson, J. (2007). Social networks and the wisdom of crowds. Network World, 24, 28–29.

    Google Scholar 

  • Kazman, R., & Chen, H. (2009). The metropolis model a new logic for development of crowdsourced systems. Communications of the ACM, 52(7), 76–84.

    Article  Google Scholar 

  • Ke, W., & Zhang, P. (2009). Motivations in open source software communities: the mediating role of effort intensity and goal commitment. International Journal of Electronic Commerce, 13(4), 39–66.

    Article  Google Scholar 

  • Ke, W., & Zhang, P. (2010). The effects of extrinsic motivations and satisfaction in open source software development. Journal of the Association for Information Systems, 11(12), 784–808.

    Google Scholar 

  • Keen, A. (2007). The cult of the amateur: How today’s internet is killing our culture and assaulting our economy. London: Nicholas Brealey.

    Google Scholar 

  • Kittur, A., Chi, E. H., & Suh, B. (2008). Crowdsourcing User Studies with Mechanical Turk. in Proceedings of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pp. 453–456.

  • Kleeman, F., Voss, G. G., & Rieder, K. (2008). Un(der)paid innovators: the commercial utilization of consumer work through crowdsourcing. Science Technology & Innovation Studies, 4(1), 5–26.

    Google Scholar 

  • Lakhani, K. R., & Wolf R. G. (2005). Why hackers do what they do: understanding motivation and effort in free/open source software projects. In J. Feller, B. Fitzgerald, S. Hissam, & K. R. Lakhani (Eds.). Perspectives on free and open source software. Cambridge, MA: MIT Press.

    Google Scholar 

  • Lakhani, K. R., Jeppesen, L. B., Lohse, P. A., & Panetta, J. A. (2007). The value of openness in scientific problem solving. Harvard Business School Working Paper, (7: 50), Retrieved October 15, from http://www.hbs.edu/research/pdf/07–050.pdf

  • Lane, S. (2010). Collective intelligence for competitive advantage: Crowd sourcing and open innovation. Continuing Education, 1–80.

  • Lee, J.-N., & Kim, Y.-G. (1999). Effect of partnership quality on is outsourcing success: conceptual framework and empirical validation. Journal of Management Information Systems, 15(4), 29–61.

    Article  Google Scholar 

  • Leimeister, J. M. (2010). Collective Intelligence. Business & Information Systems Engineering, April, pp. 245–248.

  • Leimeister, J. M., Huber, M., Bretschneider, U., & Krcmar, H. (2009). Leveraging crowdsourcing -activation-supporting components for IT-based idea competitions. Journal of Management Information Systems, 26(1), 197–224.

    Article  Google Scholar 

  • Lévy, P. (1995). Collective intelligence: Mankind’s emerging world in cyberspace (R. Bononno, Trans.). New York: Plenum.

    Google Scholar 

  • Maiolini, R., & Naggi, R. (2010). Crowdsourcing and SMEs: Opportunities and challenges. in Proceedings of ITAIS, 2010.

  • Malone, T. W., Laubacher, R., & Dellarocas, C. (2010). Harnessing crowds: Mapping the genome of collective intelligence. MIT Sloan School Working Paper, no.4732-09.

  • Marcus, A., Wu, E., & Madden, S. (2011). Crowdsourcing Databases: Query Processing with People. In Proceedings of CRDR 2011.

  • Mason, W., & Watts, D. J. (2010). Financial Incentives and the performance of crowds. SIGKDD Explorations Newsletter, 11(2), 100. ACM.

    Article  Google Scholar 

  • Mazzola, D., & Distefano, A. (2010). Crowdsourcing and the participation process for problem solving: The Case of BP. in Proceedings of ItAIS 2010 VII Conference of the Italian Chapter of AIS, pp. 42–49.

  • Meier, P. (2010). How to run a successful crowdsourcing project. Retrieved December 22, from http://irevolution.net/2010/05/05/towards-a-model-for-successful-crowdsourcing/

  • Munro, R., Steven B., Victor K., Robin M., Christopher P., Tyler S., et al. (2010). Crowdsourcing and language studies: The new generation of linguistic data. in Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Mechanical Turk, pp. 122–130.

  • Nov, O., Arazy, O., & Anderson, D. (2011). Technology-mediated citizen science participation: a motivational model. Proceedings of the AAAI International Conference on Weblogs and Social Media (ICWSM 2011), Barcelona, Spain.

  • Orlikowski, W. J., & Iacono, C. S. (2001). Desperately seeking the ‘IT’ in IS research - a call to theorizing the IT artifact. Information Systems Research, 12(2), 121–134.

    Article  Google Scholar 

  • Pettigrew, K. E., & McKechnie, L. M. (2001). The use of theory in information science research. Journal of the American Society for Information Science & Technology, 52(1), 62–73.

    Article  Google Scholar 

  • Poetz, M. K., & Schreier, M. (2009). The value of crowdsourcing: can users really compete with professionals in generating new product ideas? Proceedings of the DRUID Summer Conference, 2009, 110–128.

    Google Scholar 

  • Raddick, M., et al. (2010). Galaxy zoo: exploring the motivations of citizen science volunteers. Astronomy Education Review, 9(1).

  • Reeve, J. (2005). Understanding motivation and emotion. New York: John Wiley & Sons.

    Google Scholar 

  • Riedl, C., Blohm, I., Leimeister, J. M., & Krcmar, H. (2010). Rating scales for collective intelligence in innovation communities: Why quick and easy decision making does not get it right. In Proceedings of 2010 International Conference on Information Systems, St. Louis, Mi, USA: AIS.

  • Roberts, J. A., Hann, I. H., & Slaughter, S. A. (2006). Understanding the motivations, participation, and performance of open source software developers: a longitudinal study of the apache projects. Management Science, 52(7), 984–999.

    Google Scholar 

  • Roman, D. (2009). Crowdsourcing and the question of expertise. Communications of the ACM, 52(12), 12.

    Article  Google Scholar 

  • Rouse, A. (2010). A preliminary taxonomy of crowdsourcing. ACIS 2010 Proceedings, Paper 76.http://aisel.aisnet.org/acis2010/76.

  • Schenk, E., & Guittard, C. (2009). Crowdsourcing: What can be outsourced to the crowd, and why? Paper provided by HAL in its series of working papers with numbers halshs-00439256_v1

  • Shah, N., Dhanesha, A., & Seetharam, D. (2009). Crowdsourcing for e-governance: Case study. in Proceedings of the 3rd International Conference on Theory and Practice of Electronic Governance (New York, NY, USA, 2009), ACM, pp. 253–258.

  • Sharma, A. (2010). Crowdsourcing critical success factor model: Strategies to harness the collective intelligence of the crowd. Retrieved January 27, from http://irevolution.files.wordpress.com/2010/05/working-paper1.pdf

  • Sims, J., & Crossland, C. (2010). “Partners or Pariahs? Firm engagement with open innovation communities. in: Academy of Management Conference. Montreal, Ca.

  • Stewart, O., Huerta, J. M., & Sader, M. (2009). Designing crowdsourcing community for the enterprise. in Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 50–53.

  • Stewart, O., Lubensky, D., & Huerta, J. M. (2010). Crowdsourcing participation inequality: A SCOUT model for the enterprise domain. in Proceedings of the ACM SIGKDD Workshop on Human Computation, pp. 30–33.

  • Stolee, K. T., & Elbaum, S. (2010). Exploring the use of crowdsourcing to support empirical studies in software engineering. in Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement ACM, pp. 35–38.

  • Strauss, A. (1987). Qualitative research for social scientists. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Surowiecki, J. (2004). The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations. New York.

  • Tapscott, D., & Williams, A. D. (2007). Wikinomics: How mass collaboration changes everything. London, UK: Portfolio, Penguin.

    Google Scholar 

  • Trompette, P. (2008). Crowdsourcing as a way to access external knowledge for innovation: Control, incentive and coordination in hybrid forms of innovation. in Proceedings of 24th EGOS Colloquium, pp. 1–29.

  • von Hippel, E. (2005). Democratizing innovation. Cambridge, Mass: MIT Press.

    Google Scholar 

  • Vukovic, M. (2009). Crowdsourcing for enterprises. in Proceedings of Congress on Services, pp. 686–692.

  • Vukovic, M., Kumara, S., & Greenshpan, O. (2010). Ubiquitous crowdsourcing. in Proceedings of the 12th ACM international conference adjunct papers on Ubiquitous computing ACM, pp. 523–525.

  • Webster, J., & Watson, R. T. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly, 26(2), 13–23.

    Google Scholar 

  • Whitla, P. (2009). Crowdsourcing and its application in marketing activities. Contemporary Management Research, 5(1), 15–28.

    Google Scholar 

  • Wiggins, A., & Crowston, K. (2011). From conservation to crowdsourcing: A typology of citizen science. in Proceedings of the Forty-fourth Hawai'i International Conference on System Science (HICSS-44).

  • Wilcox, R. T. (2000). Experts and amateurs: the role of experience in internet auctions. Marketing Letters, 11, 363–374.

    Article  Google Scholar 

  • Yang, J., Adamic, L. A., & Ackerman, M. S. (2008). Crowdsourcing and knowledge Sharing: Strategic user behavior on taskcn. in Proceedings of ACM Electronic Commerce’08, pp. 246–255.

  • Zhang, P. (2008). Motivational affordances: Reasons for ICT design and USE. Communications of the ACM, 61(11), 145–147.

    Article  Google Scholar 

  • Zhang, P., & Benjamin, R. I. (2007). Understanding information related fields: a conceptual framework. Journal of the American Society for Information Science and Technology, 58(13), 1934–1947.

    Article  Google Scholar 

  • Zhang, P., & Li, N. (2005). The intellectual development of human–computer interaction research: a critical assessment of the MIS literature (1990–2002). Journal of Association for Information Systems, 6(11), 227–292.

    Google Scholar 

  • Zhang, P., Scialdone, M., & Ku, M. (2011). IT artifacts and the state of IS research. forthcoming in the Proceedings of ICIS 2011, Shanghai, China, December.

Download references

Acknowledgments

This work is jointly supported by the National Social Science Foundation in China (Grand No.10ATQ004), and the Ministry of Education, Humanities and Social Sciences Council in China (Grand No. 09YJA870014). Firstly, the authors sincerely appreciate two anonymous reviewers’ thoughtful comments and suggestions. Secondly, the authors would like to thank Prof. Ping Zhang from Syracuse University for her insightful and constructive suggestions on this paper. Thirdly, the authors thank Xu Meng from Syracuse University for his contribution during the coding stage. Finally, the authors thank the editors and publishers for their support to this article.

Author information

Authors and Affiliations

  1. School of Information Management, Nanjing University, #22 Hankou Road, Nanjing, China, 210093

    Yuxiang Zhao & Qinghua Zhu

Authors
  1. Yuxiang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qinghua Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuxiang Zhao.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhao, Y., Zhu, Q. Evaluation on crowdsourcing research: Current status and future direction. Inf Syst Front 16, 417–434 (2014). https://doi.org/10.1007/s10796-012-9350-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10796-012-9350-4

Keywords

Search

Navigation