Skip to main content
SpringerLink
Log in

Challenges in Survey Research

  • Chapter
  • First Online:
Contemporary Empirical Methods in Software Engineering

Abstract

While being an important and often used research method, survey research has been less often discussed on a methodological level in empirical software engineering than other types of research. This chapter compiles a set of important and challenging issues in survey research based on experiences with several large-scale international surveys. The chapter covers theory building, sampling, invitation and follow-up, statistical as well as qualitative analysis of survey data and the usage of psychometrics in software engineering surveys.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • AERA, APA, NCME (2014) Standards for educational and psychological testing. American Educational Research Association, Washington

    Google Scholar 

  • Amrhein V, Greenland S, McShane B (2019) Retire statistical significance. Nature 567:305–307

    Google Scholar 

  • Baltes S, Diehl S (2016) Worse than spam: issues in sampling software developers. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement, ESEM ’16. ACM, New York, pp 52:1–52:6. http://doi.acm.org/10.1145/2961111.2962628

  • Binning JF (2016) Construct. https://www.britannica.com/science/construct

  • Birks M, Mills J (2011) Grounded theory: a practical guide. Sage, Thousand Oaks

    Google Scholar 

  • Bourque P, Fairley RE et al (2014) Guide to the software engineering body of knowledge (SWEBOK): version 3.0. IEEE Computer Society Press, Washington

    Google Scholar 

  • Campbell DT, Fiske DW (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 56(2):81–105

    Google Scholar 

  • Charmaz K (2014) Constructing grounded theory. Sage, Thousand Oaks

    Google Scholar 

  • Ciolkowski M, Laitenberger O, Vegas S, Biffl S (2003) Practical experiences in the design and conduct of surveys in empirical software engineering. In: Conradi R, Wang AI (eds) Empirical methods and studies in software engineering, experiences from ESERNET, vol 2765. Lecture notes in computer science. Springer, Berlin, pp 104–128. https://doi.org/10.1007/978-3-540-45143-3_7

    Google Scholar 

  • Coaley K (2014) An introduction to psychological assessment and psychometrics. Sage, Thousand Oaks

    Google Scholar 

  • Cochran WG (1977) Sampling techniques. Wiley, New York

    MATH  Google Scholar 

  • Cohen RJ, Swerdlik ME, Phillips SM (1995) Psychological testing and assessment: an introduction to tests and measurement. Mayfield Publishing, California

    Google Scholar 

  • Cruz S, da Silva FQ, Capretz LF (2015) Forty years of research on personality in software engineering: a mapping study. Comput Hum Behav 46:94–113

    Google Scholar 

  • de Mello RM, da Silva PC, Travassos GH (2015) Investigating probabilistic sampling approaches for large-scale surveys in software engineering. J Softw Eng Res Dev 3(1):8. https://doi.org/10.1186/s40411-015-0023-0

    Google Scholar 

  • DiCiccio TJ, Efron B (1996) Bootstrap confidence intervals. Stat Sci 11(3):189–228

    MathSciNet  MATH  Google Scholar 

  • Feldt R, Torkar R, Angelis L, Samuelsson M (2008) Towards individualized software engineering: empirical studies should collect psychometrics. In: Cheng L, Sillito J, Storey MD, Tessem B, Venolia G, de Souza CRB, Dittrich Y, John M, Hazzan O, Maurer F, Sharp H, Singer, J, Sim SE (eds) Proceedings of the 2008 international workshop on cooperative and human aspects of software engineering, CHASE 2008, Leipzig. ACM, New York, pp 49–52. https://doi.org/10.1145/1370114.1370127

    Google Scholar 

  • Fowler FJ (2013) Survey research methods. Sage, Thousand Oaks

    Google Scholar 

  • Freedman D, Pisani R, Purves R (2007). Statistics. Norton, New York

    MATH  Google Scholar 

  • Ghazi AN, Petersen K, Reddy SS, Nekkanti H (2019) Survey research in software engineering: problems and mitigation strategies. IEEE Access 7:24703–24718

    Google Scholar 

  • Glaser BG (1992) Basics of grounded theory analysis: emergence vs. forcing. Sociology Press, Mill Valley

    Google Scholar 

  • Glaser BG, Strauss AL (1967) Discovery of grounded theory: strategies for qualitative research. Aldine de Gruyter, New York

    Google Scholar 

  • Graziotin D, Fagerholm F (2019) Happiness and the productivity of software engineers. In: Rethinking productivity in software engineering. Apress, Berkeley, pp 109–124

    Google Scholar 

  • Graziotin D, Wang X, Abrahamsson P (2015) Understanding the affect of developers: theoretical background and guidelines for psychoempirical software engineering. In: Proceedings of the 7th international workshop on social software engineering, SSE 2015. ACM, New York, pp 25–32. http://doi.acm.org/10.1145/2804381.2804386

    Google Scholar 

  • Graziotin D, Fagerholm F, Wang X, Abrahamsson P (2017) On the unhappiness of software developers. In: Mendes E, Counsell S, Petersen K (eds) Proceedings of the 21st international conference on evaluation and assessment in software engineering. ACM Press, New York, pp 324–333

    Google Scholar 

  • Graziotin D, Fagerholm F, Wang, Abrahamsson P (2018) What happens when software developers are (un)happy. J Syst Softw 140:32–47

    Google Scholar 

  • Gregor S (2006) The nature of theory in information systems. MIS Q 30(3):611–642. http://misq.org/the-nature-of-theory-in-information-systems.html

    Google Scholar 

  • Gren L (2018) Standards of validity and the validity of standards in behavioral software engineering research. In: Standards of validity and the validity of standards in behavioral software engineering research. ACM Press, New York

    Google Scholar 

  • Hannay JE, Sjøberg DI, DybÃ¥ T (2007) A systematic review of theory use in software engineering experiments. IEEE Trans Softw Eng 33(2):87–107

    Google Scholar 

  • Hogan R (2017) Personality and the fate of organizations. Erlbaum, Mahwah

    Google Scholar 

  • Inayat I, Salim SS, Marczak S, Daneva M, Shamshirband S (2015) A systematic literature review on agile requirements engineering practices and challenges. Comput Hum Behav 51:915–929

    Google Scholar 

  • Kalinowski M, Card DN, Travassos GH (2012) Evidence-based guidelines to defect causal analysis. IEEE Softw 29(4):16–18

    Google Scholar 

  • Kalinowski M, Curty P, Paes A, Ferreira A, Spínola RO, Fernández DM, Felderer M, Wagner S (2017) Supporting defect causal analysis in practice with cross-company data on causes of requirements engineering problems. In: Proceedings of the 39th IEEE/ACM international conference on software engineering: software engineering in practice track, ICSE-SEIP 2017, Buenos Aires. IEEE Computer Society, Silver Spring, pp 223–232. https://doi.org/10.1109/ICSE-SEIP.2017.14

  • Kass RE (2011) Statistical inference: the big picture. Stat Sci Rev J Inst Math Stat 26(1):1

    MathSciNet  Google Scholar 

  • Kasunic M (2005) Designing an effective survey. Technical report, Carnegie-Mellon University, Pittsburgh, PA and Software Engineering Institute

    Google Scholar 

  • Kitchenham BA, Pfleeger SL (2002a) Principles of survey research part 2: designing a survey. ACM SIGSOFT Softw Eng Notes 27(1):18–20. https://doi.org/10.1145/566493.566495

    Google Scholar 

  • Kitchenham BA, Pfleeger SL (2002b) Principles of survey research: part 3: constructing a survey instrument. ACM SIGSOFT Softw Eng Notes 27(2):20–24. https://doi.org/10.1145/511152.511155

    Google Scholar 

  • Kitchenham, BA, Pfleeger SL (2002c) Principles of survey research part 4: questionnaire evaluation. ACM SIGSOFT Softw Eng Notes 27(3):20–23. https://doi.org/10.1145/638574.638580

    Google Scholar 

  • Kitchenham BA, Pfleeger SL (2002d) Principles of survey research: part 5: populations and samples. ACM SIGSOFT Softw Eng Notes 27(5):17–20. https://doi.org/10.1145/571681.571686

    Google Scholar 

  • Kitchenham BA, Pfleeger SL (2008) Personal opinion surveys. In: Guide to advanced empirical software engineering. Springer, Berlin, pp 63–92

    Google Scholar 

  • Kline P (2015) A handbook of test construction (psychology revivals): introduction to psychometric design. Routledge, London

    Google Scholar 

  • Lenberg P, Feldt R, Wallgren LG (2015) Behavioral software engineering: a definition and systematic literature review. J Syst Softw 107:15–37

    Google Scholar 

  • Levine TR, Weber R, Hullett C, Park HS, Massi Lindsey LL (2008) A critical assessment of null hypothesis significance testing in quantitative communication research. Hum Commun Res 34:171–187

    Google Scholar 

  • Malhotra MK, Grover V (1998) An assessment of survey research in POM: from constructs to theory. J Oper Manag 16(4):407–425

    Google Scholar 

  • Mannio M, Nikula U (2001) Requirements elicitation using a combination of prototypes and scenarios. Technical report, Telecom Business Research Center Lappeenranta

    Google Scholar 

  • Méndez Fernández D, Passoth J-H (2018) Empirical software engineering: from discipline to interdiscipline. J Syst Softw 148:170–179

    Google Scholar 

  • Méndez Fernández D, Wagner S (2015) Naming the pain in requirements engineering: a design for a global family of surveys and first results from Germany. Inform Softw Tech 57:616–643

    Google Scholar 

  • Méndez Fernández D, Wagner S, Kalinowski M, Schekelmann A, Tuzcu A, Conte T, Spinola R, Prikladnicki R (2015) Naming the pain in requirements engineering: comparing practices in Brazil and Germany. IEEE Softw 32(5):16–23

    Google Scholar 

  • Méndez Fernández D, Wagner S, Kalinowski M, Felderer M, Mafra P, Vetrò A, Conte T, Christiansson M-T, Greer D, Lassenius C et al. (2017) Naming the pain in requirements engineering—contemporary problems, causes, and effects in practice. Empir Softw Eng 22(5):2298–2338

    Google Scholar 

  • Méndez Fernández D, Tießler M, Kalinowski M, Felderer M, Kuhrmann M (2018) On evidence-based risk management in requirements engineering. In: International conference on software quality. Springer, Berlin, pp 39–59

    Google Scholar 

  • Molléri JS, Petersen K, Mendes E (2019) CERSE-catalog for empirical research in software engineering: a systematic mapping study. Inform Softw Tech 105:117–149

    Google Scholar 

  • Pfleeger SL, Kitchenham BA (2001) Principles of survey research: part 1: turning lemons into lemonade. ACM SIGSOFT Softw Eng Notes 26(6):16–18. https://doi.org/10.1145/505532.505535

    Google Scholar 

  • Pinsonneault A, Kraemer K (1993) Survey research methodology in management information systems: an assessment. J Manag Inform Syst 10(2):75–105

    Google Scholar 

  • Pittenger DJ (1993) Measuring the MBTI…and coming up short. J Career Plan Employ 54(1):48–52

    Google Scholar 

  • Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering. Wiley, London

    Google Scholar 

  • Rust J (2009) Modern psychometrics: the science of psychological assessment. Routledge, Hove, East Sussex New York

    Google Scholar 

  • Sjøberg DI, DybÃ¥ T, Anda BC, Hannay JE (2008) Building theories in software engineering. In: Guide to advanced empirical software engineering. Springer, Berlin, pp 312–336

    Google Scholar 

  • Sommerville I, Sawyer P, Viller S (1998) Viewpoints for requirements elicitation: a practical approach. In: Proceedings of the 3rd international conference on requirements engineering (ICRE ’98), Putting requirements engineering to practice, Colorado Springs. IEEE Computer Society, Silver Spring, pp 74–81. https://doi.org/10.1109/ICRE.1998.667811

  • Stol K-J, Fitzgerald B (2015) Theory-oriented software engineering. Sci Comput Program 101:79–98

    Google Scholar 

  • Stol K, Ralph P, Fitzgerald B (2016) Grounded theory in software engineering research: a critical review and guidelines. In: Dillon LK, Visser W, Williams L (eds) Proceedings of the 38th international conference on software engineering, ICSE 2016, Austin. ACM, New York, pp 120–131. https://doi.org/10.1145/2884781.2884833

    Google Scholar 

  • Strauss A, Corbin J (1990) Basics of qualitative research. Sage, Thousand Oaks

    Google Scholar 

  • Torchiano M, Fernández DM, Travassos GH, de Mello RM (2017) Lessons learnt in conducting survey research. In: Proceedings of the 5th IEEE/ACM international workshop on conducting empirical studies in industry, CESI@ICSE 2017, Buenos Aires. IEEE, Piscataway, pp 33–39. https://doi.org/10.1109/CESI.2017.5

    Google Scholar 

  • Usman M, Britto R, Börstler J, Mendes E (2017) Taxonomies in software engineering: a systematic mapping study and a revised taxonomy development method. Inform Softw Tech 85:43–59

    Google Scholar 

  • Wagner S, Méndez Fernández D, Felderer M, Vetrò A, Kalinowski M, Wieringa R, Pfahl D, Conte T, Christiansson M-T, Greer D, Lassenius C, Männistö T, Nayebi M, Oivo M, Penzenstadler B, Prikladnicki R, Ruhe G, Schekelmann A, Sen S, Spínola R, Tuzcu A, De La Vara JL, Winkler D (2019) Status quo in requirements engineering: a theory and a global family of surveys. ACM Trans Softw Eng Methodol. 28(2):9:1–9:48

    Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, Berlin

    MATH  Google Scholar 

  • Yamane T (1973) Statistics: an introductory analysis. Longman, New York

    MATH  Google Scholar 

Download references

Acknowledgement

We are grateful to all collaborating researchers in the NaPiRE initiative.

Author information

Authors and Affiliations

  1. University of Stuttgart, Stuttgart, Germany

    Stefan Wagner & Daniel Graziotin

  2. Technical University of Munich, Munich, Germany

    Daniel Mendez

  3. Blekinge Institute of Technology, Karlskrona, Sweden

    Daniel Mendez

  4. fortiss GmbH, Munich, Germany

    Daniel Mendez

  5. Department of Computer Science, University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  6. Blekinge Institute of Technology, Karlskrona, Sweden

    Michael Felderer

  7. Pontifical Catholic University of Rio de Janeiro, Rio de Janeiro, Brazil

    Marcos Kalinowski

Authors
  1. Stefan Wagner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Mendez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Felderer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Graziotin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marcos Kalinowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Wagner .

Editor information

Editors and Affiliations

  1. Institute of Computer Science, University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Systems Engineering and Computer Science, Federal University of Rio de Janeiro, Rio de Janeiro, Rio de Janeiro, Brazil

    Guilherme Horta Travassos

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Wagner, S., Mendez, D., Felderer, M., Graziotin, D., Kalinowski, M. (2020). Challenges in Survey Research. In: Felderer, M., Travassos, G. (eds) Contemporary Empirical Methods in Software Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-32489-6_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32489-6_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32488-9

  • Online ISBN: 978-3-030-32489-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation