Abstract
We derive an analytic model of the inter-judge correlation as a function of five underlying parameters. Inter-cue correlation and the number of cues capture our assumptions about the environment, while differentiations between cues, the weights attached to the cues, and (un)reliability describe assumptions about the judges. We study the relative importance of, and interrelations between these five factors with respect to inter-judge correlation. Results highlight the centrality of the inter-cue correlation. We test the model’s predictions with empirical data and illustrate its relevance. For example, we show that, typically, additional judges increase efficacy at a greater rate than additional cues.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ariely, D., Au, W.T., Bender, R.H., Budescu, D.V., Dietz, C.B., Gu, H., Wallsten, T.S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147.
Ashton, R.H. (1986). Combining the judgments of experts: How many and which ones? Organizational Behavior and Human Decision Processes, 38, 405–414.
Ashton, A.H., & Ashton, R.H. (1985). Aggregating subjective forecasts: Some empirical results. Management Science, 31, 1499–1508.
Azen, R., & Budescu, D.V. (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8, 129–148.
Budescu, D.V. (2006). Confidence in aggregation of opinions from multiple sources. In K. Fiedler & P. Juslin (Eds.), Information sampling and adaptive cognition (pp. 327–354). Cambridge: Cambridge University Press.
Budescu, D.V., & Yu, H.T. (2007). Aggregation of opinions based on correlated cues and advisors. Journal of Behavioral Decision Making, 20, 153–177.
Clemen, R.T., & Winkler, R.L. (1985). Limits for precision and value of information from dependent sources. Operations Research, 33, 427–442.
Clemen, R.T., & Winkler, R.L. (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, 39–46.
Dawes, R.M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571–582.
Einhorn, H.J. (1974). Expert Judgment: Some necessary conditions and an example. Journal of Applied Psychology, 59, 562–571.
Hammond, K.R., & Stewart, T.R. (2001). The essential Brunswik: beginnings, explications, application. London: Oxford University Press.
Hammond, K.R., Wilkins, M.M., & Todd, F.J. (1966). A research paradigm for the study of interpersonal learning. Psychological Bulletin, 65, 221–232.
Hogarth, R.M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46.
Hursch, C.J., Hammond, K.R., & Hursch, J.L. (1964). Some methodological considerations in multiple-cue probability studies. Psychological Review, 71, 42–60.
Johnson, T.R., Budescu, D.V., & Wallsten, T.S. (2001). Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making, 14, 123–140.
Miller, S. (2008). Supporting joint human-computer judgment under uncertainty. Unpublished Dissertation at the University of Illinois at Urbana-Champaign.
Morris, P.A. (1986). Comment on Genest and Zideck’s “Combining probability distributions: A critique and annotated bibliography”. Statistical Science, 1, 141–144.
Shanteau, J. (2001). What does it mean when experts disagree? In E. Salas & G. Klein (Eds.), Linking expertise and naturalistic decision making. Earlbaum: Mahwa.
Schmidt, F.L., Johnson, R.H., & Gugel, J.F. (1978). Utility of policy capturing as an approach to graduate admissions decision making. Applied Psychological Measurement, 2, 345–357.
Wallsten, T.S., Budescu, D.V., Erev, I., & Diederich, A. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243–268.
Wallsten, T.S., & Diederich, A. (2001). Understanding pooled subjective probability estimates. Mathematical Social Sciences, 18, 1–18.
Weiss, D.J., & Shanteau, J. (2003a). The vice of consensus and the virtue of consistency. In J. Shanteau, P. Johnson, & C. Smith (Eds.), Psychological explorations of competent decision making. Cambridge: Cambridge University Press.
Weiss, D.J., & Shanteau, J. (2003b). Empirical assessment of expertise. Human Factors, 45, 104–116.
Winkler, R.L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685.
Winkler, R.L. (1981). Combining probability distributions from dependent information sources. Management Science, 27, 479–488.
Winkler, R.L., & Poses, R.M. (1993). Evaluating and combining physician’s probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543.
Yaniv, I., Choshen-Hillel, S., & Milyavsky, M. (2009). Spurious consensus and opinion revision: Why might people be more confident in their less accurate judgments? Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 558–563.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was supported by the National Science Foundation under Awards SES 02-41434 and 03-45925. The first author was supported by a grant from the National Institutes of Health under Ruth L. Kirschstein National Research Service Award PHS 2 T32 MH014257 (“Quantitative Methods for Behavioral Research”) to the University of Illinois at Urbana-Champaign.
Rights and permissions
About this article
Cite this article
Broomell, S.B., Budescu, D.V. Why Are Experts Correlated? Decomposing Correlations Between Judges. Psychometrika 74, 531–553 (2009). https://doi.org/10.1007/s11336-009-9118-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-009-9118-z