Abstract
Batteries of questions with identical response items are commonly used in survey research. This paper suggests that question order has the potential to cause systematic positive or negative bias on responses to all questions in a battery. Whilst question order effects have been studied for many decades, almost no attention has been given to this topic. The primary aim is to draw attention to this effect, to demonstrate its possible magnitude, and to discuss a range of mechanisms through which it might occur. These include satisficing, anchoring and cooperativeness. The effect seems apparent in the results of a recent survey. This was a survey of Emergency Department patients presenting to Wollongong Hospital (Australia) with apparently less urgent conditions in 2004. Two samples were taken. Question order was fixed in the first sample (n = 104; response rate RR2 = 94%), but randomised in the second sample (n = 46; response rate RR2 = 96%). Respondents were asked to indicate whether each of 18 reasons for presenting to the ED was a ‘very important reason’ a ‘moderately important reason’ or ‘not a reason’ The mean number of very important reasons selected was 56% higher in the first sample as compared to the second sample.
Similar content being viewed by others
References
Alwin D.F., Krosnick J.A. (1991). The reliability of survey attitude measurement: the influence of question and respondent attributes. Sociological Methods and Research 20:139–181
American Association for Public Opinion Research (2004) Standard Definitions: Final Disposition of Case Codes and Outcome Rates for Surveys, 3rd edn. Lenexa, Kansas: AAPOR. Available online at http://www.aapor.org (accessed July 2005).
Dillman, D. A., Smyth, J. D., Christian, L. M. & Stern, M. J. (2003). Multiple Answer Questions in Self-Administered Surveys: The Use of Check-All-That-Apply and Forced-Choice Question Formats Paper presented at the annual meeting of the American Statistical Association, San Fransisco, CA.
Duffy B. (2003). Response order effects-how do people read?. International Journal of Market Research 45: 457–475
Grice, H. P. (1975). Logic and conversation. In: P. Cole, & J. L. Morgan (eds.), Syntax and Semantics, Vol. 3. Speech Acts. New York: Academic Press, pp. 41–48.
Knauper B. (1999). The impact of age and education on response order effects in attitude measurement. Public Opinion Quarterly 63:347–370
Krosnick J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5: 213–236
Krosnick J.A., Alwin D.F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly 51: 201–219
McClendon M. J. (1986). Response order effects for dichotomous questions. Social Science Quarterly 67: 205–211
Narayan S., Krosnick J.A. (1996). Education moderates some response effects in attitude measurement. Public Opinion Quarterly 51: 201–219
Schwarz N. (1996). Cognition and Communication: Judgmental Biases, Research Methods, and the Logic of Conversation. Lawrence Erlbaum Associates, New Jersey
Schwarz, N. & Hippler, H. J. (1991). Response alternatives: the impact of their choice and presentation order. In: P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (eds.), Measurement Error in Surveys. Wiley Series in Probability and Mathematical Statistics New York: Wiley, pp. 41–46.
Siminski P., Cragg S., Middleton R., Masso M., Lago L., Green J., Eagar K. (2005). Primary care patients’ views on why they present to Emergency Departments—inappropriate attendances or inappropriate policy?. Australian Journal of Primary Health 11: 87–95
Sudman S., Bradburn N.M., Schwarz N. (1996). Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. Jossey-Bass, San Fransisco
Tourangeau R. (1984). Cognitive science and survey methods. In: Jabine T., Straf M., Tanur J., Tourangeau R. (eds). Cognitive Aspects of Survey Methodology: Building a Bridge between Disciplines. National Academy Press, Washington, DC, pp. 73–100
Tourangeau R., Rasinski K. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin 103: 299–314
Tourangeau R., Rips L.J., Rasinski K. (2000). The Psychology of Survey Response. Cambridge University Press, Cambridge
Wanke M, Schwarz N. (1997). Reducing question order effects: the operation of buffer items. In: Lyberg L.E., Biemer P.P., Collins M., de Leeuw E., Dippo C., Trewin D. (eds). Survey Measurement and Process Quality. Wiley, New York, pp. 115–139
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Siminski, P. Order Effects in Batteries of Questions. Qual Quant 42, 477–490 (2008). https://doi.org/10.1007/s11135-006-9054-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-006-9054-2