Abstract
There is extensive literature on identifying careless responses in survey data and acknowledging their negative impact on accurate data analysis (Goodman, Cryder, & Cheema, 2013; Huang et al., 2015a, b; Meade and Craig, 2012). However, there are very limited findings on how researchers can help prevent participants from responding carelessly in the first place. The current study manipulated the placement of a quality check item (i.e., early placement versus late placement) and showed that participants were less likely to carelessly respond when the quality check items were placed later in the survey. Quality check items were also able to identify careless responses more than other approaches (i.e., LongString index, completion time index, and self-reported indicators). When using quality check items, it appears to be best when used toward the end of the survey. There are plenty of opportunities for future studies to expand on this research to uncover ways that can help deter careless responding.
Similar content being viewed by others
Notes
As a reviewer suggested, I also performed an independent samples t-test to determine if there was a difference between the two conditions regarding the raw completion time (in seconds). There was not a statistically significant difference between the early condition (M = 10,991.25, SD = 40,448.07) and the late condition (M = 8260.41, SD = 35,600.41) regarding the completion time index; t(283) = 0.61, p = .55. This result is very similar to the statistically insignificant result using the completion time index.
References
Adair, J. G. (1984). The Hawthorne effect: A reconsideration of the methodological artifact. Journal of Applied Psychology, 69(2), 334–345.
Baer, R. A., Ballenger, J., Berry, D. T., & Wetter, M. W. (1997). Detection of random responding on the MMPI–A. Journal of Personality Assessment, 68(1), 139–151.
Beach, D. A. (1989). Identifying the random responder. The Journal of Psychology, 123(1), 101–103.
Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2016). Can we turn shirkers into workers? Journal of Experimental Social Psychology, 66, 20–28.
Berry, D. T., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4(3), 340–345.
Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218–229.
Chiaburu, D. S., Huang, J. L., Hutchins, H. M., & Gardner, R. G. (2014). Trainees’ perceived knowledge gain unrelated to the training domain: The joint action of impression management and motives. International Journal of Training and Development, 18(1), 37–52.
Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349–360.
Goldberg, L. R. (1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychology in Europe, 7(1), 7–28.
Goldberg, L. R., & Kilkowski, J. M. (1985). The prediction of semantic consistency in self-descriptions: Characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs. Journal of Personality and Social Psychology, 48(1), 82–98.
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.
Gordon, M. E., Slade, L. A., & Schmitt, N. (1986). The “science of the sophomore” revisited: From conjecture to empiricism. Academy of Management Review, 11(1), 191–207.
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114.
Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015a). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30(2), 299–311.
Huang, J. L., Liu, M., & Bowling, N. A. (2015b). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828–845.
Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39(1), 103–129.
Joinson, A. (1999). Social desirability, anonymity, and internet-based questionnaires. Behavior Research Methods, Instruments, & Computers, 31(3), 433–438.
Landsberger, H. (1958). Hawthorne revisited. Ithaca, New York: Cornell University.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455.
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872.
Spelke, E., Hirst, W., & Neisser, U. (1976). Skills of divided attention. Cognition, 4(3), 215–230.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Informed Consent
This study was approved by the Institutional Review Board (IRB#: IRB-FY2017–339).
Data Availability Statement
The dataset generated and analyzed during the current study are available from the author on reasonable request.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1
Appendix 2
Rights and permissions
About this article
Cite this article
Reyes, D.L. Combatting carelessness: Can placement of quality check items help reduce careless responses?. Curr Psychol 41, 6858–6866 (2022). https://doi.org/10.1007/s12144-020-01183-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12144-020-01183-4