Skip to main content
SpringerLink
Log in

Empirical software engineering experts on the use of students and professionals in experiments

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

[Context] Controlled experiments are an important empirical method to generate and validate theories. Many software engineering experiments are conducted with students. It is often claimed that the use of students as participants in experiments comes at the cost of low external validity while using professionals does not. [Objective] We believe a deeper understanding is needed on the external validity of software engineering experiments conducted with students or with professionals. We aim to gain insight about the pros and cons of using students and professionals in experiments. [Method] We performed an unconventional, focus group approach and a follow-up survey. First, during a session at ISERN 2014, 65 empirical researchers, including the seven authors, argued and discussed the use of students in experiments with an open mind. Afterwards, we revisited the topic and elicited experts’ opinions to foster discussions. Then we derived 14 statements and asked the ISERN attendees excluding the authors, to provide their level of agreement with the statements. Finally, we analyzed the researchers’ opinions and used the findings to further discuss the statements. [Results] Our survey results showed that, in general, the respondents disagreed with us about the drawbacks of professionals. We, on the contrary, strongly believe that no population (students, professionals, or others) can be deemed better than another in absolute terms. [Conclusion] Using students as participants remains a valid simplification of reality needed in laboratory contexts. It is an effective way to advance software engineering theories and technologies but, like any other aspect of study settings, should be carefully considered during the design, execution, interpretation, and reporting of an experiment. The key is to understand which developer population portion is being represented by the participants in an experiment. Thus, a proposal for describing experimental participants is put forward.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. http://isern.iese.de/

  2. http://softeng.polito.it/ESEIW2014/ISERN/program.html#Students

References

  • Basili VR, Reiter RW (May 1981) A controlled experiment quantitatively comparing software development approaches. IEEE Trans Softw Eng SE-7(3):299–320

    Article  Google Scholar 

  • Basili VR, Selby RW (Dec. 1987) Comparing the effectiveness of software testing strategies. IEEE Trans Softw Eng SE-13(12):1278–1296

    Article  Google Scholar 

  • Basili, V. R. McGarry, F. E. Pajerski, R. and Zelkowitz, M. V. 2002 “Lessons learned from 25 years of process improvement: The rise and fall of the NASA software engineering laboratory,” in Proceedings - International Conference on Software Engineering, pp. 69–79.

  • Beecham S, Hall T, Rainer A (Mar. 2003) Software process improvement problems in twelve software companies: an empirical analysis. Empir Softw Eng 8(1):7–42

    Article  MATH  Google Scholar 

  • Berander, P. 2004 “Using students as subjects in requirements prioritization,” in Proceedings - 2004 International Symposium on Empirical Software Engineering, ISESE 2004, pp. 167–176.

  • Berg K. E. and Latin, R. W. 2003 Essentials of research methods in health, Physical Education, Exercise Science, and Recreation. LWW.

  • Bergersen GR, Sjoberg DIK, Dyba T (2014) Construction and validation of an instrument for measuring programming skill. IEEE Trans Softw Eng 40(12):1163–1184

    Article  Google Scholar 

  • Bratthall, L. Johansson, E. and Regnell, B. 2000 “Is a Design Rationale Vital when Predicting Change Impact? – A Controlled Experiment on Software,” in 2nd International Conference on Product Focused Software Process Improvement (PROFES 2000),pp. 126–139.

  • Ciolkowski, M. 2009 “What do we know about perspective-based reading? An approach for quantitative aggregation in software engineering,” in 2009 3rd International Symposium on Empirical Software Engineering and Measurement, ESEM 2009, pp. 133–144.

  • Easton G, Easton A, Belch M (Sep. 2003) An experimental investigation of electronic focus groups. Inf Manag 40(8):717–727

    Article  Google Scholar 

  • Fagerholm, F. Oza, N. and Munch, J. 2013 “A platform for teaching applied distributed software development: The ongoing journey of the Helsinki software factory,” in 2013 3rd International Workshop on Collaborative Teaching of Globally Distributed Software Development, CTGDSD 2013 - Proceedings,pp. 1–5.

  • Gorschek T, Garre P, Larsson S, Wohlin C (2006) A model for technology transfer in practice. IEEE Softw 23(6):88–95

    Article  Google Scholar 

  • Höst, M. Regnell, B. and Wohlin, C. 2000 “Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment,” in Empir Softw Eng, vol. 5, pp. 201–214.

  • Jedlitschka A. and Pfahl, D. 2005 “Reporting guidelines for controlled experiments in software engineering,” in 2005 International Symposium on Empirical Software Engineering, ISESE 2005, pp. 95–104.

  • Juristo N. and Moreno, A. M. 2001 Basics of Software Engineering Experimentation, vol. 5/6.

  • Jyrki, K. Bragge, J. and Lehtola, L. 2008 “The Focus Group Method as an Empirical Tool in Software Engineering,” in Guide to Advanced Empirical Software Engineering, pp. 93–116.

  • Kontio, J. Lehtola, L. and Bragge, J. 2004 “Using the Focus Group Method in Software Engineering: Obtaining Practitioner and User Experiences,” in International Symposium on Empirical Software Engineering, pp. 271–280.

  • Lethbridge TC, Sim SE, Singer J (Jul. 2005) Studying software engineers: data collection techniques for software field studies. Empir Softw Eng 10(3):311–341

    Article  Google Scholar 

  • Lindvall, M. Basili, V. R. Boehm, B. W. Costa, P. Dangle, K. Shull, F. Tesoriero, R. Williams, L. A. and Zelkowitz, M. V. 2002“Empirical Findings in Agile Methods,” in Second XP Universe and First Agile Universe Conference on Extreme Programming and Agile Methods - XP/Agile Universe 2002,pp. 197–207.

  • M.H. and Wohlin, C. M. C. Gustavsson, A. 1996 “A Framework for Technology Introduction in Software Organizations,” in Software Process Improvement Conference,pp. 167–176.

  • McMeekin, D. A. Von Konsky, B. R. Robey, M. and Cooper, D. J. A. 2009 “The significance of participant experience when evaluating software inspection techniques,” in Proceedings of the Australian Software Engineering Conference, ASWEC, pp. 200–209.

  • Morgan DL (Aug. 1996) Focus groups. Annu Rev Sociol 22(1):129–152

    Article  Google Scholar 

  • Morgan D. L. and Krueger, R. A. 1993 “When to use focus groups and why.,” in Successful focus groups: Advancing the state of the art., pp. 3–19.

  • Porter A, Votta L (1998) Comparing detection methods for software requirements inspections: a replication using professional subjects. Empir Softw Eng 3:355–379

    Article  Google Scholar 

  • Porter AA, Votta LG, Basili VR (1995) Comparing detection methods for software requirements inspections: a replicated experiment. IEEE Trans Softw Eng 21:563–575

    Article  Google Scholar 

  • Remus, W. 1989 “Using students as subjects in experiments on decision supportsystems,” [1989] Proc. Twenty-Second Annu. Hawaii Int. Conf. Syst. Sci. Vol. III Decis. Support Knowl. Based Syst. Track, vol. 3.

  • Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2010) How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: a series of four experiments. IEEE Trans Softw Eng 36:96–118

    Article  Google Scholar 

  • Runeson, P. 2003 “Using students as experiment subjects–an analysis on graduate and freshmen student data,” Proc. 7th Int. Conf. Empir. Assess. Eval. Softw. Eng., pp. 95–102.

  • Salman, I. Misirli, A. T. and Juristo, N. 2015 “Are Students Representatives of Professionals in Software Engineering Experiments?,” in International Conference on Software Engineering (ICSE), pp. 666–676.

  • Salviulo F. and Scanniello, G. 2014 “Dealing with Identifiers and Comments in Source Code Comprehension and Maintenance: Results from an Ethnographically-informed Study with Students and Professionals,” in Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, p. 48:1--48:10.

  • Siegmund J, Siegmund N, Apel S (2015) Views on internal and external validity in empirical software engineering. In: International Conference on software engineering (ICSE)

    Google Scholar 

  • Sjoberg DIK, Anda B, Arisholm E, Dyba T, Jorgensen M, Karahasanovic A, Koren EF, Vokac M (2002) Conducting realistic experiments in software engineering. Proc. Int. Symp. Empir. Softw. Eng

  • Svahnberg, M. Aurum, A. and Wohlin, C. 2008 “Using students as subjects - an empirical evaluation,” in Proceedings of the Second ACMIEEE international symposium on Empirical software engineering and measurement, pp. 288–290.

  • Tremblay, M. Hevner, A. and Berndt, D. 2010 “Focus Groups for Artifact Refinement and Evaluation in Design Research,” Communications of the Association for Information Systems, vol. 26, no. 1.

  • Turhan B. and Bener, A. 2007 “A template for real world team projects for highly populated software engineering classes,” in Proceedings - International Conference on Software Engineering,pp. 748–751.

  • Vegas, S. Dieste, O. and Juristo, N. 2015 “Difficulties in Running Experiments in the Software Industry: Experiences from the Trenches,” in Proceedings - 3rd International Workshop on Conducting Empirical Studies in Industry, CESI 2015, pp. 3–9.

  • Wesslén A (2000) Replicated empirical study of the impact of the methods in the PSP on individual engineers. Empir Softw Eng 5:93–123

    Article  MATH  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering: an introduction. Springer

Download references

Acknowledgements

We would like to thank all the ISERN 2014 participants for the inspiring and energetic discussions. We would like to thank both the anonymous experts and the following non-anonymous experts for participating in the survey: Paris Avgeriou, Teresa Baldassarre, Victor Basili, Giovanni Cantone, Jeff Carver, Tore Dybå, Hakan Erdogmus, Vladimir Mandic, Manuel Mastrofini, Daniel Mendez, Oscar Pastor, Guilherme Horta Travassos, Stephan Wagner, Qing Wang, Roel Wieringa, and Dietmar Winkler. We thank Sonnhild Namingha for proof reading the manuscript. This research is supported in part by the Academy of Finland Project 278354.

Author information

Authors and Affiliations

  1. California Polytechnic State University, San Luis Obispo, CA, USA

    Davide Falessi

  2. Universidad Politécnica de Madrid, Madrid, Spain

    Natalia Juristo

  3. Blekinge Institute of Technology, Karlskrona, Sweden

    Claes Wohlin

  4. Brunel University London, London, UK

    Burak Turhan

  5. University of Helsinki, Helsinki, Finland

    Jürgen Münch

  6. Reutlingen University, Reutlingen, Germany

    Jürgen Münch

  7. Fraunhofer Institute for Experimental Software Engineering, Kaiserslautern, Germany

    Andreas Jedlitschka

  8. University of Oulu, Oulu, Finland

    Markku Oivo

Authors
  1. Davide Falessi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Natalia Juristo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Claes Wohlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Burak Turhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jürgen Münch
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Andreas Jedlitschka
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Markku Oivo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Davide Falessi.

Additional information

Communicated by: Mark Harman

Appendix

Appendix

1.1 Invitation Letter

Dear First name Last name,

During an ISERN 2014 session, we discussed the uses of students and professionals as subjects of software engineering experiments. Afterwards, Davide Falessi, Natalia Juristo, Claes Wohlin, Burak Turhan, Jürgen Münch, Andreas Jedlitschka, and Markku Oivo revisited the topic, elicited experts’ opinions, fostered discussion, and derived some conclusions. These conclusions are currently summarized in a paper, we aim to publish in EMSE. The aim of this research is to identify pros and cons of using students and professionals in software engineering experiments. Our results aim to support researchers during generalization of results from experiments as well as reviewers during the evaluation of experiments.

As an attendee of ISERN 2014 meeting, we care about knowing your degree of agreement with the reached conclusions. We would appreciate if you could share with us your level of agreement on our conclusions. The aim of this survey is to add the community view on our conclusions. Because we care about your opinion, you should not hesitate or be afraid to disagree with our conclusions.

The survey takes approximately 10 min to be completed. Your answers will be confidential and will only be reported in an aggregated form. Reminders will be sent to non-respondents only. You will be personally acknowledged in the paper.

As a member of ISERN, please collaborate with us by participating to this survey by using this URL:

The deadline is in two weeks from now.

If you experience any troubles or you have any comment, please contact Dr. Davide Falessi at DFalessi@calpoly.edu

Best regards,

Davide Falessi, Natalia Juristo, Claes Wohlin, Burak Turhan, Jürgen Münch, Andreas Jedlitschka, Markku Oivo.

1.2 Introduction

Thank you for participating in our survey on students and professionals in software engineering experiments.

The aim of this research is to identify pros and cons of using students and professionals in software engineering experiments. Our results aim to support researchers during generalization of results from experiments as well as reviewers during the evaluation of experiments.

The survey takes approximately 10 min to be completed. Your answers will be confidential and will only be reported in an aggregated form.

If you experience any troubles or you have any comment, please contact Dr. Davide Falessi at DFalessi@calpoly.edu

1.2.1 Experience

  1. 1.

    What is your level of experience in performing (designing, running, analyzing, reporting) controlled experiments?

  • Expert: more than 3 published experiments

  • Novice: between 1 and 2 published experiments

  • Amateur: no published experiment or you do not know what an experiment is

1.2.2 Agreement

  1. 2.

    For the following statements, we would like to get your level of agreement (completely agree, partially agree, partially disagree, completely disagree, I do not know or I do not want to answer)

  2. 1.

    Experiments with students are not of lower relevance than experiments with professionals.

  3. 2.

    Experiments with students are not of less interest than experiments with professionals.

  4. 3.

    Experiments with students might exhibit lower external validity than experiments with professionals.

  5. 4.

    Internal and external validities are of equal importance.

  6. 5.

    Classifying experiment participants using a binary scale (students or professional) is an inappropriate approach.

  7. 6.

    Students participating in an experiment are a convenience sampling.

  8. 7.

    Professionals participating in an experiment are a convenient sampling.

  9. 8.

    We should think about population and validity already before conducting the experiment at the time when we are planning to use convenience sampling.

  10. 9.

    The use of students better supports the improvement of experiment design and protocol than professionals.

  11. 10.

    Conducting experiments with professionals as a first step should not be encouraged unless high sample sizes are guaranteed.

  12. 11.

    Conducting experiments with professionals entails a higher treatment conformance threat to validity than experiments with students.

  13. 12.

    Conducting experiments with professionals entails longer learning cycles than experiments with students.

  14. 13.

    Conducting experiments with professionals underestimate the positive effects of new technology than experiments with students.

  15. 14.

    The suitability and representativeness of students as proxies for professional developers change with different contexts and with different types of population.

1.3 Thank You

We appreciate the time and effort you spent in answering this survey. Please report your last name if you want to be personally acknowledged in the paper.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Falessi, D., Juristo, N., Wohlin, C. et al. Empirical software engineering experts on the use of students and professionals in experiments. Empir Software Eng 23, 452–489 (2018). https://doi.org/10.1007/s10664-017-9523-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9523-3

Keywords

Search

Navigation