Abstract
Citizens should have an appropriate level of trust in experts—neither too low nor too high. Experts cannot guarantee accuracy in a complex world. But they can attempt to be well calibrated, meaning that they clearly communicate their confidence in their knowledge, and that confidence is lower in domains where their accuracy is likely to be lower. I review research on expert calibration, on the effects of confidence and calibration on perceived credibility, and on the role that “naive realism” plays in biasing our assessments of experts who say what we want to hear.
Prepared for the Nebraska Symposium on Motivation. I thank Brian Bornstein, John Campbell, Dan Kahan, and Saul Perlmutter for helpful conversations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Routledge Dictionary of Latin Quotations, 2004, p. 107.
- 2.
- 3.
This is quite similar to the definition given by Mayer et al. (1995, p. 712): “…the willingness of a party to be vulnerable to the actions of another party based on the expectation that the other will perform a particular action important to the trustor, irrespective of the ability to monitor or control that other party.” But the latter phrase seems unnecessarily restrictive; the phrase “trust but verify” suggests that we often seek to monitor those we have entrusted with a task (see Williamson, 1993).
- 4.
- 5.
Ioannidis (2005) famously argues that the percentage of reported findings in the literature that are true might be quite low (anywhere from 85 % down to .15 % in his simulations) depending on various assumptions about typical statistical power, the prior probability of our hypotheses, and the nature and direction of biases in our research methods.
- 6.
The adversarial motive is nonepistemic, but experts can have nonepistemic motives (e.g., to make money) without caring whether they win. In such cases, it is often their sponsor who wants to win.
- 7.
A technicality: At the bottom of the confidence scale, one can never be overconfident, and at the top, one can never be underconfident—but from the consumer’s standpoint, the source is still overconfident. And overconfidence can be observed in datasets that are not vulnerable to this problem (e.g., Brenner, Koehler, Liberman, & Tversky, 1996).
- 8.
When my colleagues and I published a study demonstrating why the effects of marijuana legalization on use and revenues were extremely uncertain (Kilmer, Caulkins, Pacula, MacCoun, & Reuter, 2010), we were denounced on various websites for being either useless or cowardly.
- 9.
This also implies that an overconfident expert is unfairly restricting the decision maker’s zone of discretion.
- 10.
Given the complexity and stochastic nature of many causal systems, I think it is probably theoretically possible for two experts, each well calibrated in the past, to be fairly confident in opposing predictions, but only under rare circumstances. In the three-dimensional space of confidence, calibration, and disagreement, that corner mostly will be empty.
- 11.
Another relevant literature looks at expert testimony at trial (Cutler & Kovera, 2011).
- 12.
The IEM can be found at https://tippie.uiowa.edu/iem/; the 2014 Senate trading is summarized at https://tippie.uiowa.edu/iem/media/story.cfm?ID=3389
References
Allum, N., Sturgis, P., Tabourazi, D., & Brunton-Smith, I. (2008). Science knowledge and attitudes across cultures: a meta-analysis. Public Understanding of Science, 17, 35–54.
Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S., Birch, S., Birt, A. R., et al. (2014). Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.
Angrist, J. D., & Pischke, J. S. (2010). The credibility revolution in empirical economics: how better research design is taking the con out of econometrics. Journal of Economic Perspectives, 24, 3–30.
Arrow, K. J., et al. (2008). The promise of prediction markets. Science, 320, 877–878.
Berezow, A. B., & Campbell, H. (2012). Science left behind: Feel-good fallacies and the rise of the anti-scientific left. New York, NY: PublicAffairs.
Bornmann, L., & Mungra, P. (2011). Improving peer review in scholarly journals. European Science Editing, 37, 41–43.
Boykoff, M. T., & Boykoff, J. M. (2004). Balance as bias: global warming and the US prestige press. Global Environmental Change, 14, 125–136.
Bradfield, A. L., & Wells, G. L. (2000). The perceived validity of eyewitness identification testimony: A test of the five Biggers criteria. Law and Human Behavior, 24, 581–594.
Braun, P. A., & Yaniv, I. (1992). A case study of expert judgment: Economists’ probabilities versus base-rate model forecasts. Journal of Behavioral Decision Making, 5, 217–231.
Braver, S. L., Thoemmes, F. J., & Rosenthal, R. (2014). Continuously cumulating meta-analysis and replicability. Perspectives on Psychological Science, 9, 333–342.
Brenner, L. A., Koehler, D. J., Liberman, V., & Tversky, A. (1996). Overconfidence in probability and frequency judgments: A critical examination. Organizational Behavior and Human Decision Processes, 65, 212–219.
Carlisle, J. E., Feezell, J. T., Michaud, K. E., Smith, E. R., & Smith, L. (2010). The public’s trust in scientific claims regarding offshore oil drilling. Public Understanding of Science, 19, 514–527.
Commons, M. L., Miller, P. M., & Gutheil, T. G. (2004). Expert witness perceptions of bias in experts. Journal of the American Academy of Psychiatry and the Law, 32, 70–75.
Cosmides, L., & Tooby, J. (1992). Cognitive adaptations for social exchange. In J. Barkow, L. Cosmides, & J. Tooby (Eds.), The adapted mind: Evolutionary psychology and the generation of culture. New York, NY: Oxford University Press.
Critchley, C. R. (2008). Public opinion and trust in scientists: The role of the research context, and the perceived motivation of stem cell researchers. Public Understanding of Science, 17, 309–327.
Cronin, E. B., & Sugimoto, C. R. (2014). Beyond bibliometrics: Multidimensional indicators of scholarly impact. Cambridge, MA: MIT Press.
Cutler, B. L., & Kovera, M. B. (2011). Expert psychological testimony. Current Directions in Psychological Science, 20, 53–57.
Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668–1674.
Eagly, A. H., Wood, W., & Chaiken, S. (1978). Causal inferences about communicators and their effect on opinion change. Journal of Personality and Social Psychology, 36, 424–435.
Fanelli, D., & Ioannidis, J. P. A. (2013). US studies may overestimate effect sizes in softer research. Proceedings of the National Academy of Science, 110, 15031–15036.
Fink, W., Lipatov, V., & Konitzer, M. (2009). Diagnoses by general practitioners: Accuracy and reliability. International Journal of Forecasting, 25, 784–793.
French, J. P. R., Jr., & Raven, B. (1960). The bases of social power. In D. Cartwright & A. Zander (Eds.), Group dynamics (pp. 607–623). New York, NY: Harper and Row.
Gauchat, G. (2012). Politicization of science in the public sphere: A study of public trust in the United States, 1974 to 2010. American Sociological Review, 77, 167–187.
Giles, J. (2002). Scientific wagers: Wanna bet? Nature, 420, 354–355.
Gilovich, T., Griffin, D., & Kahneman, D. (2002). Heuristics and biases: The psychology of intuitive judgment. Cambridge: Cambridge University Press.
Goodman-Delahunty, J., Granhag, P. A., Hartwig, M., & Loftus, E. F. (2010). Insightful or wishful: Lawyers’ ability to predict case outcomes. Psychology, Public Policy, and Law, 16, 133–157.
Grice, P. (1989). Studies in the ways of words. Cambridge MA: Harvard University Press.
Henrion, M., & Fischhoff, B. (1986). Assessing uncertainty in physical constants. American Journal of Physics, 54, 791–798.
Hmielowski, J. D., Feldman, L., Myers, T. A., Leiserowitz, A., & Maibach, E. (2014) An attack on science? Media use, trust in scientists, and perceptions of global warming. Public Understanding of Science, 23, 866–883.
Hovland, C. I., Janis, I. L., & Kelley, J. J. (1953). Communication and persuasion. New Haven, CT: Yale University Press.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2, 696–701.
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23, 524–532.
Jones, E. E., & Harris, V. A. (1967). The attribution of attitudes. Journal of Experimental Psychology, 3, 1–24.
Kahan, D. M. (2013). Ideology, motivated reasoning, and cognitive reflection. Judgment and Decision Making, 8, 407–424.
Kahan, D. M., Jenkins-Smith, H., & Braman, D. (2011). Cultural cognition of scientific consensus. Journal of Risk Research, 14, 147–174.
Kahan, D. M., Braman, D., Slovic, P., Gastil, J., & Cohen, G. (2012). Cultural cognition of the risks and benefits of nanotechnology. Nature Nanotechnology, 4, 87–90.
Kahan, D. M., Peters, E., Dawson, E. C., & Slovic, P. (2013). Motivated numeracy and enlightened self-government. New Haven, CT: Yale Law School. Public Law Working Paper No. 307. Retrieved from http://ssrn.com/abstract=2319992.
Kahan, D. M., Peters, E., Wittlin, M., Slovic, P., Larrimore Oullette, L., Braman, D., et al. (2012). The polarizing impact of science literacy and numeracy on perceived climate change. Nature Climate Change, 2, 732–735.
Kilmer, B., Caulkins, J. P., Pacula, R. L., MacCoun, R. J., & Reuter, P. H. (2010). Altered state? Assessing how marijuana legalization in California could influence marijuana consumption and public budgets. Santa Monica, CA: RAND.
Kramer, R. M. (1999). Trust and distrust in organizations: Emerging perspectives, enduring questions. Annual Review of Psychology, 50, 69–98.
Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480–498.
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: the state of the art to 1980. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.
Lillienfeld, S. O. (2012). Public skepticism of psychology: Why many people perceive the study of human behavior as unscientific. American Psychologist, 67, 111–129.
Lin, S. W., & Bier, V. M. (2008). A study of expert overconfidence. Reliability Engineering & System Safety, 93, 711–721.
Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: the effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37, 2098–2109.
MacCoun, R. (1998). Biases in the interpretation and use of research results. Annual Review of Psychology, 49, 259–287.
MacCoun, R. J., & Paletz, S. (2009). Citizens’ perceptions of ideological bias in research on public policy controversies. Political Psychology, 30, 43–65.
MacCoun, R. J., & Perlmutter, S. (in press). Blind analysis as a correction for confirmatory bias in physics and in psychology. In Lilienfeld, S. O., & Waldman, I. (Eds.), Psychological science under scrutiny: Recent challenges and proposed solutions. John Wiley and Sons.
Makridakis, S., Hogarth, R. M., & Gaba, A. (2009). Forecasting and uncertainty in the economic arena. International Journal of Forecasting, 25, 794–812.
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational trust. Academy of Management Review, 20, 709–734.
McKenzie, C. R. M., Liersch, M. J., & Yaniv, I. (2008). Overconfidence in interval estimates: What does expertise buy you? Organizational Behavior and Human Decision Processes, 107, 179–191.
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., et al. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science, 25, 1106–1115.
Merton, R. K. (1938). Science and the social order. Philosophy of Science, 5, 321–337.
Michaels, D., & Monforton, C. (2005). Manufacturing uncertainty: Contested science and the protection of the public’s health and environment. American Journal of Public Health, 95, S39–S48.
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., et al. (2014). Promoting transparency in social science research. Science, 343, 30–31.
Mooney, C. (2006). The Republican war on science. New York, NY: Basic Books.
Mooney, C. (2012). The Republican brain: The science of why they deny science and reality. New York, NY: Wiley.
Moore, D. A., Cain, D. M., Loewenstein, G. and Bazerman, M. (eds.). (2005). Conflicts of interest: Problems and solutions from law, medicine and organizational settings. London: Cambridge University Press.
Murphy, A. H., & Winkler, R. L. (1977). Reliability of subjective probability forecasts of precipitation and temperature. Journal of the Royal Statistical Society: Applied Statistics, 26, 41–47.
Murrie, D. C., Boccaccini, M. T., Guarnera, L. A., & Rufino, K. A. (2013). Are forensic experts biased by the side that retained them? Psychological Science , 24, 1889–1897.
NSF. (2014). Science and engineering indicators 2014. Washington, DC: National Science Foundation.
Oreskes, N., & Conway, E. M. (2010). Merchants of doubt: How a handful of scientists obscured the truth on issues from tobacco smoke to global warming. New York, NY: Bloomsbury Press.
Pasher, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528–530.
Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. New York, NY: Springer-Verlag.
Pew Research. (2009). Public praises science; scientists fault public, media. Washington, DC: Pew Research Center for the People and the Press.
Price, P. C., & Stone, E. R. (2004). I ntuitive evaluation of likelihood judgment producers: Evidence for a confidence heuristic. Journal of Behavioral Decision Making, 17, 39–57.
Pronin, E., Gilovich, T., & Ross, L. (2004). Objectivity in the eye of the beholder: Divergent perceptions of bias in self vs. others. Psychological Review, 111, 781–799.
Retzbach, A., & Maier, M. (2014). Communicating scientific uncertainty: Media effects on public engagement with science. Communication Research. [print version not out yet]
Rotter, J. B., & Stein, D. K. (1971). Public attitudes toward the trustworthiness, competence, and altruism of twenty selected occupations. Journal of Applied Social Psychology, 1, 334–343.
Rousseau, D. M., Sitkin, S. B., Burt, R. S., & Camerer, C. (1998). Not so different after all: A cross-discipline view of trust. Academy of Management Review, 23, 393–404.
Sah, S., Moore, D., & MacCoun, R. (2013). Cheap talk and credibility: The consequences of confidence and accuracy on advisor credibility and persuasiveness. Organizational Behavior and Human Decision Processes, 121, 246–255.
Satterfield, T., Kandlikar, M., Beaudrie, C. E. H., Conti, J., & Harthorn, B. H. (2009). Anticipating the perceived risk of nanotechnologies. Nature Nanotechnology, 4, 752–758.
Silver, N. (2012). The signal and the noise. New York, NY: Penguin.
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1875–1888.
Tenney, E. R., MacCoun, R. J., Spellman, B. A., & Hastie, R. (2007). Calibration trumps confidence as a basis for witness credibility. Psychological Science, 18, 46–50.
Tenney, E. R., Spellman, B. A., & MacCoun, R. J. (2008). The benefits of knowing what you know (and what you don’t): Fact-finders rely on others who are well calibrated. Journal of Experimental Social Psychology, 44, 1368–1375.
Tetlock, P. E. (2005). Expert political judgment: How good is it? How can we know? Princeton, NJ: Princeton University Press.
Tetlock, P. E., Mellers, B. A., Rohrbaugh, N., & Chen, E. (2014). Forecasting tournaments: Tools for increasing transparency and improving the quality of debate. Current Directions in Psychological Science, 23, 290–295.
Thibaut, J., & Walker, L. (1978). A theory of procedure. California Law Review, 66, 541–566.
Twenge, J. M., Campbell, W. K., & Carter, N. T. (2014). Declines in trust in others and confidence in institutions among American adults and late adolescents, 1972−2012. Psychological Science, 25, 1914–1923.
Wakeman, N. (2011, February 11). IBM’s ‘Jeopardy!’ match more than game playing: Big Blue sees Watson driving strides in analytics. Washington Technology.
Walster, E., Aronson, E., & Abrahams, D. (1966). On increasing the persuasiveness of a low prestige communicator. Journal of Experimental Social Psychology, 2, 325–342.
Williamson, O. E. (1993). Calculativeness, trust, and economic organization. Journal of Law and Economics, 36, 453–486.
Wood, W., Jones, M., & Benjamin, L. T. (1986). Surveying psychology’s public image. American Psychologist, 41, 947–953.
Yong, E. (2012). Bad copy. Nature, 485, 298–300.
Zahavi, A., & Zahavi, A. (1997). The handicap principle. Oxford: Oxford University Press.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
MacCoun, R.J. (2015). The Epistemic Contract: Fostering an Appropriate Level of Public Trust in Experts. In: Bornstein, B., Tomkins, A. (eds) Motivating Cooperation and Compliance with Authority. Nebraska Symposium on Motivation, vol 62. Springer, Cham. https://doi.org/10.1007/978-3-319-16151-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-319-16151-8_9
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16150-1
Online ISBN: 978-3-319-16151-8
eBook Packages: Behavioral ScienceBehavioral Science and Psychology (R0)