Skip to main content
SpringerLink
Log in

On proper scoring rules and cumulative prospect theory

  • Original Article
  • Published:
EURO Journal on Decision Processes

Abstract

Scoring rules are traditional techniques to measure the association between a reported belief and an observed outcome. The condition that a scoring rule is proper means that an agent maximizes his expected score when he reports a belief that equals his true belief. The implicit assumption that the agent is risk neutral is, however, often unrealistic, at least when the underlying agent is a human. Modern decision theories based on rank-dependent utilities, such as cumulative prospect theory, have been shown to be more effective at describing how human beings make decisions under risk and uncertainty. Traditional proper scoring rules are, however, incompatible with cumulative prospect theory because they fail to satisfy a property called comonotonicity. In this paper, we provide novel insights on why comonotonicity is crucial to make proper scoring rules indeed proper when eliciting beliefs from cumulative prospect theory agents. After suggesting strategies to create comonotonic proper scoring rules, we propose calibration procedures to obtain an agent’s true belief by removing the influence of the agent’s value function and weighting functions from his reported belief, when beliefs are elicited by means of comonotonic proper scoring rules.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Abdellaoui M (2000) Parameter-free elicitation of utility and probability weighting functions. Manage Sci 46(11):1497–1512

    Article  Google Scholar 

  • Abdellaoui M, Bleichrodt H, L’Haridon O (2008) A tractable method to measure utility and loss aversion under prospect theory. J Risk Uncert 36(3):245–266

    Article  Google Scholar 

  • Allais M (1953) Violations of the betweenness axiom and nonlinearity in probability. Econometrica 21:503–546

    Article  Google Scholar 

  • Allen F (1987) Discovering personal probabilities when utility functions are unknown. Manage Sci 33(4):542–544

    Article  Google Scholar 

  • Armantier O, Treich N (2013) Eliciting beliefs: proper scoring rules, incentives, stakes and hedging. Eur Econ Rev 62:17–40

    Article  Google Scholar 

  • Barberis N, Xiong W (2009) What drives the disposition effect? An analysis of a long-standing preference-based explanation. J Fin 64(2):751–784

    Article  Google Scholar 

  • Bickel JE (2010) Scoring rules and decision analysis education. Decis Anal 7(4):346–357

    Article  Google Scholar 

  • Birnbaum MH (1997) Violations of monotonicity in judgment and decision making. In: Marley AAJ (ed) Choice, decision, and measurement: essays in Honor of R. Duncan Luce, pp 73–100

  • Birnbaum MH (2008) New paradoxes of risky decision making. Psychol Rev 115(2):463–501

    Article  Google Scholar 

  • Camerer CF (2004) Prospect theory in the wild: evidence from the field. In: Camerer CF, Loewenstein G, Rabin M (eds) Advances in behavioral economics, chapter 5, pp 148–161

  • Carvalho A (2015) Tailored proper scoring rules elicit decision weights. Judgm Decis Mak 10(1):86–96

    Google Scholar 

  • Carvalho A (2016a) A note on Sandroni–Shmaya belief elicitation mechanism. J Predict Mark 10(2):14–21

    Google Scholar 

  • Carvalho A (2016b) An overview of applications of proper scoring rules. Decis Anal 13(4):223–242

    Article  Google Scholar 

  • Carvalho A (2017) On a participation structure that ensures representative prices in prediction markets. Decis Support Syst 104:13–25

    Article  Google Scholar 

  • Carvalho A, Larson K (2012) Sharing rewards among strangers based on peer evaluations. Decis Anal 9(3):253–273

    Article  Google Scholar 

  • Chambers CP (2008) Proper scoring rules for general decision models. Games Econ Behav 63(1):32–40

    Article  Google Scholar 

  • Friedman D (1983) Effective scoring rules for probabilistic forecasts. Manage Sci 29(4):447–454

    Article  Google Scholar 

  • Gilboa I (1987) Expected utility with purely subjective non-additive probabilities. J Math Econ 16(1):65–88

    Article  Google Scholar 

  • Gneiting T, Raftery AE (2007) Strictly Proper scoring rules, prediction, and estimation. J Am Stat Assoc 102(477):359–378

    Article  Google Scholar 

  • Gonzalez R, Wu G (1999) On the shape of the probability weighting function. Cogn Psychol 38:129–166

    Article  Google Scholar 

  • Hanson R (2003) Combinatorial information market design. Inf Syst Front 5(1):107–119

    Article  Google Scholar 

  • Hines G, Larson K (2010) Preference elicitation for risky prospects. In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, pp 889–896

  • Hossain T, Okui R (2013) The binarized scoring rule. Rev Econ Stud 80(3):984–1001

    Article  Google Scholar 

  • Johnstone DJ, Jose VRR, Winkler RL (2011) Tailored scoring rules for probabilities. Decis Anal 8(4):256–268

    Article  Google Scholar 

  • Kadane JB, Winkler RL (1988) Separating probability elicitation from utilities. J Am Stat Assoc 83(402):357–363

    Article  Google Scholar 

  • Karni E (2009) A mechanism for eliciting probabilities. Econometrica 77(2):603–606

    Article  Google Scholar 

  • Kothiyal A, Spinu V, Wakker PP (2011) Comonotonic proper scoring rules to measure ambiguity and subjective beliefs. J Multi-Crit Decis Anal 17(3–4):101–113

    Google Scholar 

  • Nakazono Y (2013) Strategic behavior of federal open market committee board members: evidence from members’ forecasts. J Econ Behav Organ 93:62–70

    Article  Google Scholar 

  • Offerman T, Palley AB (2016) Lossed in translation: an off-the-shelf method to recover probabilistic beliefs from loss-averse agents. Exp Econ 19(1):1–30

    Article  Google Scholar 

  • Offerman T, Sonnemans J, Van De Kuilen G, Wakker PP (2009) A truth serum for Non-Bayesians: correcting proper scoring rules for risk attitudes. Rev Econ Stud 76(4):1461–1489

    Article  Google Scholar 

  • Parkes DC, Wellman MP (2015) Economic reasoning and artificial intelligence. Science 349(6245):267–272

    Article  Google Scholar 

  • Perny P, Viappiani P, Boukhatem A (2016) Incremental preference elicitation for decision making under risk with the rank-dependent utility model. Uncert Artif Intell

  • Quiggin J (1982) A theory of anticipated utility. J Econ Behav Organ 3(4):323–343

    Article  Google Scholar 

  • Sandroni A, Shmaya E (2013) Eliciting beliefs by paying in chance. Econ Theory Bull 1(1):33–37

    Article  Google Scholar 

  • Savage LJ (1971) Elicitation of personal probabilities and expectations. J Am Stat Assoc 66(336):783–801

    Article  Google Scholar 

  • Schervish MJ (1989) A general method for comparing probability assessors. Ann Stat 17(4):1856–1879

    Article  Google Scholar 

  • Schlag KH, Tremewan J, Van der Weele JJ (2015) A penny for your thoughts: a survey of methods for eliciting beliefs. Exp Econ 3(1):457–490

    Article  Google Scholar 

  • Schlag KH, van der Weele JJ (2013) Eliciting probabilities, means, medians, variances and covariances without assuming risk neutrality. Theor Econ Lett 3(1):38–42

    Article  Google Scholar 

  • Schmeidler D (1989) Subjective probability and expected utility without additivity. Econometrica 57(3):571–587

    Article  Google Scholar 

  • Schotter A, Trevino I (2014) Belief elicitation in the laboratory. Annu Rev Econ 6:103–128

    Article  Google Scholar 

  • Selten R, Sadrieh A, Abbink K (1999) Money does not induce risk neutral behavior, but binary lotteries do even worse. Theor Decis 46(3):211–249

    Article  Google Scholar 

  • Spiegelhalter DJ (1986) Probabilistic prediction in patient management and clinical trials. Stat Med 5(5):421–433

    Article  Google Scholar 

  • Starmer C (2000) Developments in non-expected utility theory: the hunt for a descriptive theory of choice under risk. J Econ Lit 38(2):332–382

    Article  Google Scholar 

  • Tetlock PE (2005) Expert political judgment: how good is it? How can we know? Princeton University Press

  • Toda M (1963) Measurement of subjective probability distributions. Technical Report ESD-TDR-63-407, Decision Sciences Laboratory, Electronic Systems Division, Air Force Systems Command, United States Air Force, Bedford, MA

  • Tversky A, Kahneman D (1992) Advances in prospect theory: cumulative representation of uncertainty. J Risk Uncert 5(4):297–323

    Article  Google Scholar 

  • Wakker P, Deneffe D (1996) Eliciting von Neumann–Morgenstern utilities when probabilities are distorted or unknown. Manage Sci 42(8):1131–1150

    Article  Google Scholar 

  • Wakker PP (2010) Prospect theory: for risk and ambiguity. Cambridge university press

  • Weber BJ, Chapman GB (2005) Playing for peanuts: why is risk seeking more common for low-stakes gambles? Organ Behav Hum Decis Process 97(1):31–46

    Article  Google Scholar 

  • Winkler RL, Murphy AH (1968) “Good” probability assessors. J Appl Meteorol 7(5):751–758

    Article  Google Scholar 

  • Winkler RL, Murphy AH (1970) Nonlinear utility and the probability score. J Appl Meteorol 9:143–148

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank Jonathan Baron and two anonymous reviewers for useful comments. The authors also thank the Natural Sciences and Engineering Research Council of Canada for funding this research.

Author information

Authors and Affiliations

  1. Department of Information Systems and Analytics, Farmer School of Business, Miami University, Oxford, OH, 45056, USA

    Arthur Carvalho

  2. Department of Management Sciences, University of Waterloo, Waterloo, ON, N2L 3G1, Canada

    Stanko Dimitrov

  3. David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, N2L 3G1, Canada

    Kate Larson

Authors
  1. Arthur Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stanko Dimitrov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kate Larson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arthur Carvalho.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Carvalho, A., Dimitrov, S. & Larson, K. On proper scoring rules and cumulative prospect theory. EURO J Decis Process 6, 343–376 (2018). https://doi.org/10.1007/s40070-018-0081-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40070-018-0081-8

Keywords

Mathematics Subject Classification

Search

Navigation